elm.web.file_loader.AsyncWebFileLoader

class AsyncWebFileLoader(header_template=None, verify_ssl=True, aget_kwargs=None, pw_launch_kwargs=None, pdf_read_kwargs=None, html_read_kwargs=None, pdf_read_coroutine=None, html_read_coroutine=None, pdf_ocr_read_coroutine=None, file_cache_coroutine=None, browser_semaphore=None, use_scrapling_stealth=False, num_pw_html_retries=3, **__)[source]

Bases: BaseAsyncFileLoader

Async web file (PDF or HTML) loader

Purpose:

Save content from links as files.

Responsibilities:

  1. Retrieve data from a URL.

  2. Determine wether information should be stored as a PDF or HTML document.

Key Relationships:

Returns either PDFDocument or HTMLDocument. Uses aiohttp to access the web.

Parameters:

  • header_template (dict, optional) – Optional GET header template. If not specified, uses DEFAULT_HEADERS. By default, None.

  • verify_ssl (bool, optional) – Option to use aiohttp’s default SSL check. If False, SSL certificate validation is skipped. By default, True.

  • aget_kwargs (dict, optional) – Other kwargs to pass to aiohttp.ClientSession.get(). By default, None.

  • pw_launch_kwargs (dict, optional) – Keyword-value argument pairs to pass to async_playwright.chromium.launch() (only used when reading HTML). By default, None.

  • pdf_read_kwargs (dict, optional) – Keyword-value argument pairs to pass to the pdf_read_coroutine. By default, None.

  • html_read_kwargs (dict, optional) – Keyword-value argument pairs to pass to the html_read_coroutine. By default, None.

  • pdf_read_coroutine (callable, optional) – PDF file read coroutine. Must by an async function. Should accept PDF bytes as the first argument and kwargs as the rest. Must return a elm.web.document.PDFDocument. If None, a default function that runs in the main thread is used. By default, None.

  • html_read_coroutine (callable, optional) – HTML file read coroutine. Must by an async function. Should accept HTML text as the first argument and kwargs as the rest. Must return a elm.web.document.HTMLDocument. If None, a default function that runs in the main thread is used. By default, None.

  • pdf_ocr_read_coroutine (callable, optional) – PDF OCR file read coroutine. Must by an async function. Should accept PDF bytes as the first argument and kwargs as the rest. Must return a elm.web.document.PDFDocument. If None, PDF OCR parsing is not attempted, and any scanned PDF URL’s will return a blank document. By default, None.

  • file_cache_coroutine (callable, optional) – File caching coroutine. Can be used to cache files downloaded by this class. Must accept an Document instance as the first argument and the file content to be written as the second argument. If this method is not provided, no document caching is performed. By default, None.

  • browser_semaphore (asyncio.Semaphore, optional) – Semaphore instance that can be used to limit the number of playwright browsers open concurrently. If None, no limits are applied. By default, None.

  • use_scrapling_stealth (bool, default=False) – Option to use scrapling stealth scripts instead of tf-playwright-stealth. By default, False.

  • num_pw_html_retries (int, default=3) – Number of attempts to load HTML content. This is useful because the playwright parameters are stochastic, and sometimes a combination of them can fail to load HTML. The default value is likely a good balance between processing attempts and retrieval success. Note that the minimum number of attempts will always be 2, even if the user provides a value smaller than this. By default, 3.

Methods

fetch(source)

Fetch a document for the given source.

fetch_all(*sources)

Fetch documents for all requested sources.

Attributes

PAGE_LOAD_TIMEOUT

Default page load timeout value in milliseconds
PAGE_LOAD_TIMEOUT = 60000

Default page load timeout value in milliseconds

async fetch(source)

Fetch a document for the given source.

Parameters:

source (str) – Source used to load the document.

Returns:

elm.web.document.Document – Document instance containing text, if the load was successful.

async fetch_all(*sources)

Fetch documents for all requested sources.

Parameters:

*sources – Iterable of sources (as strings) used to fetch the documents.

Returns:

list – List of documents, one per requested sources.