jade.models.spark.SparkConfigModel

class jade.models.spark.SparkConfigModel(*, collect_worker_logs: bool = False, conf_dir: str = 'spark-conf', container: SparkContainerModel, enabled: bool = False, master_node_memory_overhead_gb: int = 3, node_memory_overhead_gb: int = 10, run_user_script_inside_container: bool = False, use_tmpfs_for_scratch: bool = False, alt_scratch: str = None, worker_memory_gb: int = 0)

Bases: JadeBaseModel

Model definition for a Spark configuration

Create a new model by parsing and validating input data from keyword arguments.

Raises ValidationError if the input data cannot be parsed to form a valid model.

Methods

construct([_fields_set])	Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data.
copy(*[, include, exclude, update, deep])	Duplicate a model, optionally choose which fields to include, exclude and change.
dict(*[, include, exclude, by_alias, ...])	Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.
from_orm(obj)
get_run_user_script()
get_spark_script()
get_start_history_server()
get_start_master()
get_start_worker(memory, cluster)
get_stop_history_server()
get_stop_master()
get_stop_worker()
handle_legacy_values(values)
json(*[, include, exclude, by_alias, ...])	Generate a JSON representation of the model, include and exclude arguments as per dict().
load(path)	Load a model from a file.
parse_file(path, *[, content_type, ...])
parse_obj(obj)
parse_raw(b, *[, content_type, encoding, ...])
schema([by_alias, ref_template])
schema_json(*[, by_alias, ref_template])
update_forward_refs(**localns)	Try to update ForwardRefs on fields based on this Model, globalns and localns.
validate(value)

Attributes

collect_worker_logs
conf_dir
container
enabled
master_node_memory_overhead_gb
node_memory_overhead_gb
run_user_script_inside_container
use_tmpfs_for_scratch
alt_scratch
worker_memory_gb

classmethod construct(_fields_set: SetStr | None = None, **values: Any) → Model

Creates a new model setting __dict__ and __fields_set__ from trusted or pre-validated data. Default values are respected, but no other validation is performed. Behaves as if Config.extra = ‘allow’ was set since it adds all passed values

copy(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, update: DictStrAny | None = None, deep: bool = False) → Model

Duplicate a model, optionally choose which fields to include, exclude and change.

Parameters:

• include – fields to include in new model

• exclude – fields to exclude from new model, as with values this takes precedence over include

• update – values to change/add in the new model. Note: the data is not validated before creating the new model: you should trust this data

• deep – set to True to make a deep copy of the model

Returns:

new model instance

dict(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, by_alias: bool = False, skip_defaults: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False) → DictStrAny

Generate a dictionary representation of the model, optionally specifying which fields to include or exclude.

json(*, include: AbstractSetIntStr | MappingIntStrAny | None = None, exclude: AbstractSetIntStr | MappingIntStrAny | None = None, by_alias: bool = False, skip_defaults: bool | None = None, exclude_unset: bool = False, exclude_defaults: bool = False, exclude_none: bool = False, encoder: Callable[[Any], Any] | None = None, models_as_dict: bool = True, **dumps_kwargs: Any) → str

Generate a JSON representation of the model, include and exclude arguments as per dict().

encoder is an optional function to supply as default to json.dumps(), other arguments as per json.dumps().

classmethod load(path: Path)

Load a model from a file.

classmethod update_forward_refs(**localns: Any) → None

Try to update ForwardRefs on fields based on this Model, globalns and localns.