elm.ords.services.threaded.StoreFileOnDisk

class StoreFileOnDisk(out_dir, tpe_kwargs=None)[source]

Bases: ThreadedService

Abstract service that manages the storage of a file on disk.

Storage can occur due to creation or a move of a file.

Parameters:
  • out_dir (path-like) – Path to output directory where file should be stored.

  • tpe_kwargs (dict, optional) – Keyword-value argument pairs to pass to concurrent.futures.ThreadPoolExecutor. By default, None.

Methods

acquire_resources()

Open thread pool and temp directory

call(*args, **kwargs)

Call the service.

process(doc)

Store file in out directory.

process_using_futures(fut, *args, **kwargs)

Process a call to the service.

release_resources()

Shutdown thread pool and cleanup temp directory

Attributes

MAX_CONCURRENT_JOBS

Max number of concurrent job submissions.

can_process

Always True (limiting is handled by asyncio)

name

Service name used to pull the correct queue object.

property can_process

Always True (limiting is handled by asyncio)

Type:

bool

async process(doc)[source]

Store file in out directory.

Parameters:

doc (elm.web.document.Document) – Document containing meta information about the file. Must have relevant processing keys in the metadata dict, otherwise the file may not be stored in the output directory.

Returns:

Path | None – Path to output file, or None if no file was stored.

MAX_CONCURRENT_JOBS = 10000

Max number of concurrent job submissions.

acquire_resources()

Open thread pool and temp directory

async classmethod call(*args, **kwargs)

Call the service.

Parameters:

*args, **kwargs – Positional and keyword arguments to be passed to the underlying service processing function.

Returns:

obj – A response object from the underlying service.

property name

Service name used to pull the correct queue object.

Type:

str

async process_using_futures(fut, *args, **kwargs)

Process a call to the service.

Parameters:
  • fut (asyncio.Future) – A future object that should get the result of the processing operation. If the processing function returns answer, this method should call fut.set_result(answer).

  • **kwargs – Keyword arguments to be passed to the underlying processing function.

release_resources()

Shutdown thread pool and cleanup temp directory