gaps.legacy.BatchJob

class BatchJob(config)

Bases: BatchJob

Legacy reV-like batch job framework.

To use this class, simply override the following two attributes:

PIPELINE_CLASSPipeline

Pipeline class with at least two class methods: run and cancel_all The run method must take pipeline_config, monitor, and verbose as arguments.

PIPELINE_BACKGROUND_METHODcallable

Callable to run pipeline in the background with monitoring. If you set this as a static method (e.g. PIPELINE_BACKGROUND_METHOD = staticmethod(my_callable)), then this function should take exactly two arguments: pipeline_config and verbose. Otherwise, the function should take three arguments, where the first is a reference to this batch class, and the last two are the same as above.

Parameters:

config (str) – File path to config json or csv (str).

Methods

cancel()	Cancel all pipeline modules for all batch jobs.
cancel_all(config[, verbose])	Cancel all reV pipeline modules for all batch jobs.
delete()	Clear all of the batch sub job folders.
delete_all(config[, verbose])	Delete all reV batch sub job folders based on the job summary csv in the batch config directory.
run(config[, dry_run, delete, ...])	Run the reV batch job from a config file.

Attributes

PIPELINE_BACKGROUND_METHOD	Formattable string of the base pipeline CLI command.
PIPELINE_CLASS	Formattable string of the base pipeline CLI command.
job_table	Batch job summary table.
sub_dirs	Job sub directory paths.

abstract property PIPELINE_CLASS

Formattable string of the base pipeline CLI command.

Type:

str

abstract property PIPELINE_BACKGROUND_METHOD

Formattable string of the base pipeline CLI command.

Type:

str

classmethod cancel_all(config, verbose=False)

Cancel all reV pipeline modules for all batch jobs.

Parameters:

• config (str) – File path to batch config json or csv (str).

• verbose (bool) – Flag to turn on debug logging.

classmethod delete_all(config, verbose=False)

Delete all reV batch sub job folders based on the job summary csv in the batch config directory.

Parameters:

• config (str) – File path to batch config json or csv (str).

• verbose (bool) – Flag to turn on debug logging.

classmethod run(config, dry_run=False, delete=False, monitor_background=False, verbose=False)

Run the reV batch job from a config file.

Parameters:

• config (str) – File path to config json or csv (str).

• dry_run (bool) – Flag to make job directories without running.

• delete (bool) – Flag to delete all batch job sub directories based on the job summary csv in the batch config directory.

• monitor_background (bool) – Flag to monitor all batch pipelines continuously in the background using the nohup command. Note that the stdout/stderr will not be captured, but you can set a pipeline “log_file” to capture logs.

• verbose (bool) – Flag to turn on debug logging for the pipelines.

cancel()

Cancel all pipeline modules for all batch jobs.

delete()

Clear all of the batch sub job folders.

Only the batch sub folders listed in the job summary csv file in the batch config directory are deleted.

property job_table

Batch job summary table.

Type:

pd.DataFrame

property sub_dirs

Job sub directory paths.

Type:

list