jade.spark.metrics.SparkMetrics

class jade.spark.metrics.SparkMetrics(manager_node, history=False)[source]

Bases: object

Records metrics from a Spark cluster

Methods

generate_metrics(output_dir)

Generate metrics from a Spark cluster into files at the given path.

list_applications()

Return the applications in the cluster.

Attributes

METRICS_FILE

generate_metrics(output_dir: Path)[source]

Generate metrics from a Spark cluster into files at the given path.

Parameters:

  • cluster (str) – Cluster master node name

  • output_dir (Path) – Output directory in which to write files

list_applications()[source]

Return the applications in the cluster.

Returns:

List of application objects

Return type:

list