jade.jobs.job_queue.JobQueue

class jade.jobs.job_queue.JobQueue(max_queue_depth, existing_jobs=None, poll_interval=10, monitor_func=None, monitor_interval=10)

Bases: object

Submits jobs for execution in parallel.

There are two ways to use this class:

1. Build a list of jobs and pass that list to JobQueue.run_jobs(). It will run to completion.

2. Call JobQueue.submit() as jobs become ready to run. JobQueue will either run it immediately or queue it if too many commands are outstanding. In this mode it is up to the caller to call JobQueue.process_queue() periodically. That will look for job completions pull new jobs off the queue. JobQueue does not start a background thread to do this automatically.

Parameters:

• max_queue_depth (int) – Maximum number of sub-processes to maintain

• poll_interval (int) – Seconds to sleep in between completion checks.

• monitor_interval (int) – Seconds to sleep in between resource monitor cycles. Can be overridden with environment variable JADE_MONITOR_INTERVAL.

Methods

is_full()	Return True if the max number of jobs is outstanding.
process_queue()	Process completions and submit new jobs if the queue is not full.
run(jobs)	Run job queue synchronously.
run_jobs(jobs, max_queue_depth[, ...])	Run job queue synchronously.
submit(job)	Submit a job to be executed.
wait()	Return once all jobs have completed.

Attributes

outstanding_jobs

Return the outstanding jobs.

is_full()

Return True if the max number of jobs is outstanding.

Return type:

bool

property outstanding_jobs

Return the outstanding jobs.

Return type:

dict_values

process_queue()

Process completions and submit new jobs if the queue is not full.

run(jobs)

Run job queue synchronously. Blocks until all jobs are complete.

Parameters:

jobs (list) – List of AsyncJobInterface objects to run.

submit(job)

Submit a job to be executed. If the queue is not full then it will run the job. Otherwise, it will queue the job. Returns immediately. The caller should call JobQueue.process_queue() periodically to check for completions and start new jobs.

Parameters:

job (AsyncJobInterface)

wait()

Return once all jobs have completed.

classmethod run_jobs(jobs, max_queue_depth, poll_interval=10, monitor_func=None, monitor_interval=10)

Run job queue synchronously. Blocks until all jobs are complete.

Parameters:

• jobs (list) – List of AsyncJobInterface objects

• max_queue_depth (int) – Maximum number of parallel jobs to maintain

• poll_interval (int) – Seconds to sleep in between completion checks.

• monitor_func (callable) – Optionally a function to call each poll_interval, such as for resource monitoring.

• monitor_interval (int) – Interval in seconds on which to run monitor_func.