jade.extensions.generic_command.generic_command_configuration.GenericCommandConfiguration

class jade.extensions.generic_command.generic_command_configuration.GenericCommandConfiguration(**kwargs)[source]

Bases: JobConfiguration

A class used to configure generic_command jobs.

Init GenericCommand class

Parameters:
  • kwargs

  • arguments (extra)

Methods

add_job(job)

Add a job to the configuration.

add_user_data(key, data)

Add user data referenced by a key.

append_submission_group(submission_group)

Append a submission group.

assign_default_submission_group(submitter_params)

auto_config(inputs[, ...])

Create a configuration from all available inputs.

check_job_dependencies()

Check for impossible conditions with job dependencies.

check_job_estimated_run_minutes(group_name)

Check that estimated_run_minutes is set for all jobs in a group.

check_job_runtimes()

Check for any job with a longer estimated runtime than the walltime.

check_spark_config()

If Spark jobs are present in the config, configure the params to run one job at a time.

check_submission_groups()

Check for invalid job submission group assignments.

clear()

Clear all configured jobs.

create_from_result(job, output_dir)

Create an instance from a result file.

deserialize(filename_or_data[, ...])

Create a class instance from a saved configuration file.

dump([filename, stream, indent])

Convert the configuration to structured text format.

dumps([fmt_module])

Dump the configuration to a formatted string.

get_default_submission_group()

Return the default submission group.

get_job(name)

Return the job matching name.

get_num_jobs()

Return the number of jobs in the configuration.

get_submission_group(name)

Return the submission group matching name.

get_user_data(key)

Get the user data associated with key.

iter_jobs()

Yields a generator over all jobs.

job_execution_class(extension_name)

Return the class used for job execution.

job_parameters_class(extension_name)

Return the class used for job parameters.

list_jobs()

Return a list of all jobs.

list_user_data_keys()

List the stored user data keys.

reconfigure_jobs(jobs)

Reconfigure with a list of jobs.

remove_job(job)

Remove a job from the configuration.

remove_user_data(key)

Remove the key from the user data config.

serialize([include])

Create data for serialization.

serialize_for_execution(scratch_dir[, ...])

Serialize config data for efficient execution.

serialize_jobs(directory)

Serializes main job data to job-specific files.

show_jobs()

Show the configured jobs.

shuffle_jobs()

Shuffle the job order.

Attributes

FILENAME_DELIMITER

FORMAT_VERSION

job_global_config

Return the global configs applied to all jobs.

node_setup_command

Command to run on each node before starting jobs

node_teardown_command

Command to run on each node after completing jobs

setup_command

Command to run by submitter before submitting jobs

submission_groups

Return the submission groups.

teardown_command

Command to run by last node before completing jobs

classmethod auto_config(inputs, cancel_on_blocking_job_failure=False, minutes_per_job=None, append_job_name=False, append_output_dir=False, **kwargs)[source]

Create a configuration from all available inputs.

add_job(job)[source]

Add a job to the configuration.

Parameters:

job (JobParametersInterface)

create_from_result(job, output_dir)[source]

Create an instance from a result file.

Parameters:
Return type:

class

add_user_data(key, data)

Add user data referenced by a key. Must be JSON-serializable

Parameters:
  • key (str)

  • data (any)

Raises:

InvalidParameter – Raised if the key is already stored.

append_submission_group(submission_group)

Append a submission group.

Parameters:

submission_group (SubmissionGroup)

check_job_dependencies()

Check for impossible conditions with job dependencies.

Raises:

InvalidConfiguration – Raised if job dependencies have an impossible condition.

check_job_estimated_run_minutes(group_name)

Check that estimated_run_minutes is set for all jobs in a group.

check_job_runtimes()

Check for any job with a longer estimated runtime than the walltime.

Raises:

InvalidConfiguration – Raised if any job is too long.

check_spark_config()

If Spark jobs are present in the config, configure the params to run one job at a time.

check_submission_groups()

Check for invalid job submission group assignments. Make a default group if none are defined and assign it to each job.

Raises:

InvalidConfiguration – Raised if submission group assignments are invalid.

clear()

Clear all configured jobs.

classmethod deserialize(filename_or_data, do_not_deserialize_jobs=False)

Create a class instance from a saved configuration file.

Parameters:
  • filename (str | dict) – path to configuration file or that file loaded as a dict

  • do_not_deserialize_jobs (bool) – Set to True to avoid the overhead of loading all jobs from disk. Job_names will be stored instead of jobs.

Return type:

class

Raises:

InvalidParameter – Raised if the config file has invalid parameters.

dump(filename=None, stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'>, indent=2)

Convert the configuration to structured text format.

Parameters:
  • filename (str | None) – Write configuration to this file (must be .json or .toml). If None, write the text to stream. Recommend using .json for large files. .toml is much slower.

  • stream (file) – File-like interface that supports write().

  • indent (int) – If JSON, use this indentation.

Raises:

InvalidParameter – Raised if filename does not have a supported extenstion.

dumps(fmt_module=<module 'toml' from '/opt/hostedtoolcache/Python/3.9.19/x64/lib/python3.9/site-packages/toml/__init__.py'>, **kwargs)

Dump the configuration to a formatted string.

get_default_submission_group()

Return the default submission group.

Return type:

SubmissionGroup

get_job(name)

Return the job matching name.

Return type:

namedtuple

get_num_jobs()

Return the number of jobs in the configuration.

Return type:

int

get_submission_group(name)

Return the submission group matching name.

Parameters:

name (str)

Return type:

SubmissionGroup

get_user_data(key)

Get the user data associated with key.

Parameters:

key (str)

Return type:

any

iter_jobs()

Yields a generator over all jobs.

Yields:

iterator over JobParametersInterface

job_execution_class(extension_name)

Return the class used for job execution.

Parameters:

extension_name (str)

Return type:

class

property job_global_config

Return the global configs applied to all jobs.

job_parameters_class(extension_name)

Return the class used for job parameters.

Parameters:

extension_name (str)

Return type:

class

list_jobs()

Return a list of all jobs.

Returns:

list of JobParametersInterface

Return type:

list

list_user_data_keys()

List the stored user data keys.

Returns:

list of str

Return type:

list

property node_setup_command

Command to run on each node before starting jobs

property node_teardown_command

Command to run on each node after completing jobs

reconfigure_jobs(jobs)

Reconfigure with a list of jobs.

Parameters:

DistributionConfiguration.parameter_type (list of)

remove_job(job)

Remove a job from the configuration.

Parameters:

job (JobParametersInterface)

remove_user_data(key)

Remove the key from the user data config.

Parameters:

key (str)

serialize(include=ConfigSerializeOptions.JOBS)

Create data for serialization.

serialize_for_execution(scratch_dir, are_inputs_local=True)

Serialize config data for efficient execution.

Parameters:
  • scratch_dir (str) – Temporary storage space on the local system.

  • are_inputs_local (bool) – Whether the existing input data is local to this system. For many configurations accessing the input data across the network by many concurrent workers can cause a bottleneck and so implementations may wish to copy the data locally before execution starts. If the storage access time is very fast the question is irrelevant.

Returns:

Name of serialized config file in scratch directory.

Return type:

str

serialize_jobs(directory)

Serializes main job data to job-specific files.

Parameters:

directory (str)

property setup_command

Command to run by submitter before submitting jobs

show_jobs()

Show the configured jobs.

shuffle_jobs()

Shuffle the job order.

property submission_groups

Return the submission groups.

Return type:

list

property teardown_command

Command to run by last node before completing jobs