compass.services.threaded.HTMLFileLoader

class HTMLFileLoader(**kwargs)

Bases: ThreadedService

Service that loads HTML files from disk

Parameters:

**kwargs – Keyword-value argument pairs to pass to concurrent.futures.ThreadPoolExecutor. By default, None.

Methods

acquire_resources()	Open thread pool and temp directory
call(*args, **kwargs)	Call the service
process(html_fp, **kwargs)	Read HTML file from disk
process_using_futures(fut, *args, **kwargs)	Process a call to the service
release_resources()	Shutdown thread pool and cleanup temp directory

Attributes

MAX_CONCURRENT_JOBS	Max number of concurrent job submissions.
can_process	True because can always read file
name	Service name used to pull the correct queue object

property can_process

True because can always read file

Type:

bool

async process(html_fp, **kwargs)

Read HTML file from disk

Parameters:

• html_fp (path-like) – Path to HTML file on disk.

• **kwargs – Additional keyword-value argument pairs to pass to elm.web.document.HTMLDocument.

MAX_CONCURRENT_JOBS = 10000

Max number of concurrent job submissions.

acquire_resources()

Open thread pool and temp directory

async classmethod call(*args, **kwargs)

Call the service

Parameters:

• *args – Positional and keyword arguments to be passed to the underlying service processing function.

• **kwargs – Positional and keyword arguments to be passed to the underlying service processing function.

Returns:

object – A response object from the underlying service.

property name

Service name used to pull the correct queue object

Type:

str

async process_using_futures(fut, *args, **kwargs)

Process a call to the service

Parameters:

• fut (asyncio.Future) – A future object that should get the result of the processing operation. If the processing function returns answer, this method should call fut.set_result(answer).

• **kwargs – Keyword arguments to be passed to the underlying processing function.

release_resources()

Shutdown thread pool and cleanup temp directory