compass.scripts.download.download_known_urls#

async download_known_urls(jurisdiction, urls, browser_semaphore=None, file_loader_kwargs=None)[source]#

Download documents from known URLs

Parameters:

  • jurisdiction (Jurisdiction) – Jurisdiction instance representing the jurisdiction corresponding to the documents.

  • urls (Iterable of str) – Collection of URLs to download documents from.

  • browser_semaphore (asyncio.Semaphore, optional) – Semaphore instance that can be used to limit the number of downloads happening concurrently. If None, no limits are applied. By default, None.

  • file_loader_kwargs (dict, optional) – Dictionary of keyword arguments pairs to initialize elm.web.file_loader.AsyncFileLoader. By default, None.

Returns:

out_docs (list) – List of BaseDocument instances containing documents from the URL’s, or an empty list if something went wrong during the retrieval process.

Notes

Requires TempFileCachePB service to be running.

On this page