elm.ords.services.base.RateLimitedService

class RateLimitedService(rate_limit, rate_tracker)

Bases: Service

Abstract Base Class representing a rate-limited service (e.g. OpenAI)

Parameters:

• rate_limit (int | float) – Max usage per duration of the rate tracker. For example, if the rate tracker is set to compute the total over minute-long intervals, this value should be the max usage per minute.

• rate_tracker (elm.ords.utilities.usage.TimeBoundedUsageTracker) – A TimeBoundedUsageTracker instance. This will be used to track usage per time interval and compare to rate_limit.

Methods

acquire_resources()	Use this method to allocate resources, if needed
call(*args, **kwargs)	Call the service.
process(*args, **kwargs)	Process a call to the service.
process_using_futures(fut, *args, **kwargs)	Process a call to the service.
release_resources()	Use this method to clean up resources, if needed

Attributes

MAX_CONCURRENT_JOBS	Max number of concurrent job submissions.
can_process	Check if usage is under the rate limit.
name	Service name used to pull the correct queue object.

MAX_CONCURRENT_JOBS = 10000

Max number of concurrent job submissions.

acquire_resources()

Use this method to allocate resources, if needed

async classmethod call(*args, **kwargs)

Call the service.

Parameters:

*args, **kwargs – Positional and keyword arguments to be passed to the underlying service processing function.

Returns:

obj – A response object from the underlying service.

property can_process

Check if usage is under the rate limit.

property name

Service name used to pull the correct queue object.

Type:

str

abstract async process(*args, **kwargs)

Process a call to the service.

Parameters:

*args, **kwargs – Positional and keyword arguments to be passed to the underlying processing function.

async process_using_futures(fut, *args, **kwargs)

Process a call to the service.

Parameters:

• fut (asyncio.Future) – A future object that should get the result of the processing operation. If the processing function returns answer, this method should call fut.set_result(answer).

• **kwargs – Keyword arguments to be passed to the underlying processing function.

release_resources()

Use this method to clean up resources, if needed