Torc API

Functions to access the Torc Database API

torc.api.make_api(database_url) → DefaultApi

Instantiate an OpenAPI client object from a database URL.

torc.api.iter_documents(func: Callable, *args, skip=0, **kwargs) → Generator[Any, None, None]

Return a generator of documents where the API service employs batching.

Parameters:

func – API function

Yields:

OpenAPI [pydantic] model or dict, depending on what the API function returns

torc.api.send_api_command(func, *args, raise_on_error=True, timeout=120, **kwargs) → Any

Send an API command while tracking time, if timer_stats_collector is enabled.

Parameters:

  • func (function) – API function

  • args (arguments to forward to func)

  • raise_on_error (bool) – Raise an exception if there is an error, defaults to True.

  • timeout (float) – Timeout in seconds

  • kwargs (keyword arguments to forward to func)

Raises:

  • ApiException – Raised for errors detected by the server.

  • DatabaseOffline – Raised for all connection errors.

torc.api.add_jobs(api: DefaultApi, workflow_key: str, jobs, max_transfer_size=10000) → list[JobModel]

Add an iterable of jobs to the workflow.

Parameters:

  • api (DefaultApi)

  • workflow_key (str)

  • jobs (list) – Any iterable of JobModel

  • max_transfer_size (int) – Maximum number of jobs to add per API call. 10,000 is recommended.

Returns:

List of keys of created jobs. Provided in same order as jobs.

Return type:

list

torc.api.map_function_to_jobs(api: DefaultApi, workflow_key, module: str, func: str, params: list[dict], postprocess_func: str | None = None, module_directory=None, resource_requirements=None, scheduler=None, start_index=1, name_prefix='', blocked_by: list[str] | None = None) → list[JobModel]

Add a job that will call func for each item in params.

Parameters:

  • api (DefaultApi)

  • workflow_key (str)

  • module (str) – Name of module that contains func. If it is not available in the Python path, specify the parent directory in module_directory.

  • func (str) – Name of the function in module to be called.

  • params (list[dict]) – Each item in this list will be passed to func. The contents must be serializable to JSON.

  • postprocess_func (str) – Optional name of the function in module to be called to postprocess all results.

  • module_directory (str | None) – Required if module is not importable.

  • resource_requirements (str | None) – Optional id of resource_requirements that should be used by each job.

  • scheduler (str | None) – Optional id of scheduler that should be used by each job.

  • start_index (int) – Starting index to use for job names.

  • name_prefix (str) – Prepend job names with this prefix; defaults to an empty string. Names will be the index converted to a string.

  • blocked_by (None | list[str]) – Job IDs that should block all jobs created by this function.

Return type:

list[JobModel]