How Tos

Contents:

• Getting Started
  • Installation
  • Deploy sparkctl in an HPC environment
    • URLs
    • sparkctl configuration file
• Configuration
  • How to select compute nodes
  • Heterogeneous Slurm jobs
    • Interactive job
    • Batch job
  • How to handle compute node failures
  • How to use a custom spark-default.conf file
  • How to set a custom Spark log level
• Execution
  • Start a Spark Cluster
  • How to run Spark jobs in Python
    • Interactive session with pyspark
    • Jupyter notebook
    • Script execution with spark-submit
  • How to to monitor Spark resource utilization
    • Managed execution
• Applications
  • How to configure a Hive metastore
  • Visualize Data with Tableau
    • Concepts
    • Compute Node Instructions
    • Client-side Instructions
    • Persistent metastore and spark-warehouse
• Debugging
  • Spark web UI
  • Log files
  • Spark shuffle partitions
  • Slow local storage
  • Executors are spilling to disk
  • Data skew
  • Performance monitoring
    • Automated resource monitoring
    • Spark tuning resources