jade.extensions.demo.autoregression_configuration.AutoRegressionConfiguration¶

class jade.extensions.demo.autoregression_configuration.AutoRegressionConfiguration(container=None, job_global_config=None, job_post_process_config=None, user_data=None, submission_groups=None, setup_command=None, teardown_command=None, node_setup_command=None, node_teardown_command=None, **kwargs)[source]¶

Bases: JobConfiguration

A class used to configure auto-regression jobs

Constructs JobConfiguration.

Parameters:

inputs (JobInputsInterface)
container (JobContainerInterface)

Methods

`add_job`(job)	Add a job to the configuration.
`add_user_data`(key, data)	Add user data referenced by a key.
`append_submission_group`(submission_group)	Append a submission group.
`assign_default_submission_group`(submitter_params)
`check_job_dependencies`()	Check for impossible conditions with job dependencies.
`check_job_estimated_run_minutes`(group_name)	Check that estimated_run_minutes is set for all jobs in a group.
`check_job_runtimes`()	Check for any job with a longer estimated runtime than the walltime.
`check_spark_config`()	If Spark jobs are present in the config, configure the params to run one job at a time.
`check_submission_groups`()	Check for invalid job submission group assignments.
`clear`()	Clear all configured jobs.
`create_from_result`(job, output_dir)	Create an instance from a result file.
`deserialize`(filename_or_data[, ...])	Create a class instance from a saved configuration file.
`dump`([filename, stream, indent])	Convert the configuration to structured text format.
`dumps`([fmt_module])	Dump the configuration to a formatted string.
`get_default_submission_group`()	Return the default submission group.
`get_job`(name)	Return the job matching name.
`get_num_jobs`()	Return the number of jobs in the configuration.
`get_submission_group`(name)	Return the submission group matching name.
`get_user_data`(key)	Get the user data associated with key.
`iter_jobs`()	Yields a generator over all jobs.
`job_execution_class`(extension_name)	Return the class used for job execution.
`job_parameters_class`(extension_name)	Return the class used for job parameters.
`list_jobs`()	Return a list of all jobs.
`list_user_data_keys`()	List the stored user data keys.
`reconfigure_jobs`(jobs)	Reconfigure with a list of jobs.
`remove_job`(job)	Remove a job from the configuration.
`remove_user_data`(key)	Remove the key from the user data config.
`serialize`([include])	Create data for serialization.
`serialize_for_execution`(scratch_dir[, ...])	Serialize config data for efficient execution.
`serialize_jobs`(directory)	Serializes main job data to job-specific files.
`show_jobs`()	Show the configured jobs.
`shuffle_jobs`()	Shuffle the job order.

Attributes

`FILENAME_DELIMITER`
`FORMAT_VERSION`
`job_global_config`	Return the global configs applied to all jobs.
`node_setup_command`	Command to run on each node before starting jobs
`node_teardown_command`	Command to run on each node after completing jobs
`setup_command`	Command to run by submitter before submitting jobs
`submission_groups`	Return the submission groups.
`teardown_command`	Command to run by last node before completing jobs

create_from_result(job, output_dir)[source]¶

Create an instance from a result file.

Parameters:

job (JobParametersInterface)
output_dir (str)

Return type:

class

add_job(job)¶

Add a job to the configuration.

Parameters:: job (JobParametersInterface)

add_user_data(key, data)¶

Add user data referenced by a key. Must be JSON-serializable

Parameters:

key (str)
data (any)

Raises:

InvalidParameter – Raised if the key is already stored.

append_submission_group(submission_group)¶

Append a submission group.

Parameters:: submission_group (SubmissionGroup)

check_job_dependencies()¶

Check for impossible conditions with job dependencies.

Raises:: InvalidConfiguration – Raised if job dependencies have an impossible condition.

check_job_estimated_run_minutes(group_name)¶: Check that estimated_run_minutes is set for all jobs in a group.

check_job_runtimes()¶

Check for any job with a longer estimated runtime than the walltime.

Raises:: InvalidConfiguration – Raised if any job is too long.

check_spark_config()¶: If Spark jobs are present in the config, configure the params to run one job at a time.

check_submission_groups()¶

Check for invalid job submission group assignments. Make a default group if none are defined and assign it to each job.

Raises:: InvalidConfiguration – Raised if submission group assignments are invalid.

clear()¶: Clear all configured jobs.

classmethod deserialize(filename_or_data, do_not_deserialize_jobs=False)¶

Create a class instance from a saved configuration file.

Parameters:

filename (str | dict) – path to configuration file or that file loaded as a dict
do_not_deserialize_jobs (bool) – Set to True to avoid the overhead of loading all jobs from disk. Job_names will be stored instead of jobs.

Return type:

class

Raises:

InvalidParameter – Raised if the config file has invalid parameters.

dump(filename=None, stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, indent=2)¶

Convert the configuration to structured text format.

Parameters:

filename (str | None) – Write configuration to this file (must be .json or .toml). If None, write the text to stream. Recommend using .json for large files. .toml is much slower.
stream (file) – File-like interface that supports write().
indent (int) – If JSON, use this indentation.

Raises:

InvalidParameter – Raised if filename does not have a supported extenstion.

dumps(fmt_module=<module 'toml' from '/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/toml/__init__.py'>, **kwargs)¶: Dump the configuration to a formatted string.

get_default_submission_group()¶

Return the default submission group.

Return type:: SubmissionGroup

get_job(name)¶

Return the job matching name.

Return type:: namedtuple

get_num_jobs()¶

Return the number of jobs in the configuration.

Return type:: int

get_submission_group(name)¶

Return the submission group matching name.

Parameters:: name (str)
Return type:: SubmissionGroup

get_user_data(key)¶

Get the user data associated with key.

Parameters:: key (str)
Return type:: any

iter_jobs()¶

Yields a generator over all jobs.

Yields:: iterator over JobParametersInterface

job_execution_class(extension_name)¶

Return the class used for job execution.

Parameters:: extension_name (str)
Return type:: class

property job_global_config¶: Return the global configs applied to all jobs.

job_parameters_class(extension_name)¶

Return the class used for job parameters.

Parameters:: extension_name (str)
Return type:: class

list_jobs()¶

Return a list of all jobs.

Returns:: list of JobParametersInterface
Return type:: list

list_user_data_keys()¶

List the stored user data keys.

Returns:: list of str
Return type:: list

property node_setup_command¶: Command to run on each node before starting jobs

property node_teardown_command¶: Command to run on each node after completing jobs

reconfigure_jobs(jobs)¶

Reconfigure with a list of jobs.

Parameters:: DistributionConfiguration.parameter_type (list of)

remove_job(job)¶

Remove a job from the configuration.

Parameters:: job (JobParametersInterface)

remove_user_data(key)¶

Remove the key from the user data config.

Parameters:: key (str)

serialize(include=ConfigSerializeOptions.JOBS)¶: Create data for serialization.

serialize_for_execution(scratch_dir, are_inputs_local=True)¶

Serialize config data for efficient execution.

Parameters:

scratch_dir (str) – Temporary storage space on the local system.
are_inputs_local (bool) – Whether the existing input data is local to this system. For many configurations accessing the input data across the network by many concurrent workers can cause a bottleneck and so implementations may wish to copy the data locally before execution starts. If the storage access time is very fast the question is irrelevant.

Returns:

Name of serialized config file in scratch directory.

Return type:

str

serialize_jobs(directory)¶

Serializes main job data to job-specific files.

Parameters:: directory (str)

property setup_command¶: Command to run by submitter before submitting jobs

show_jobs()¶: Show the configured jobs.

shuffle_jobs()¶: Shuffle the job order.

property submission_groups¶

Return the submission groups.

Return type:: list

property teardown_command¶: Command to run by last node before completing jobs