compass.scripts.download.download_known_urls#

async download_known_urls(jurisdiction, urls, browser_semaphore=None, file_loader_kwargs=None)[source]#

Download documents from known URLs

Parameters:
  • jurisdiction (Jurisdiction) – Jurisdiction instance representing the jurisdiction corresponding to the documents.

  • urls (iterable of str) – Collection of URLs to download documents from.

  • browser_semaphore (asyncio.Semaphore, optional) – Semaphore instance that can be used to limit the number of downloads happening concurrently. If None, no limits are applied. By default, None.

  • file_loader_kwargs (dict, optional) – Dictionary of keyword arguments pairs to initialize elm.web.file_loader.AsyncFileLoader. By default, None.

Returns:

out_docs (list) – List of BaseDocument instances containing documents from the URL’s, or an empty list if something went wrong during the retrieval process.

Notes

Requires TempFileCachePB service to be running.