Slurm Workflows

This document explains how Torc simplifies running workflows on Slurm-based HPC systems. The key insight is that you don’t need to understand Slurm schedulers or workflow actions to run workflows on HPC systems—Torc handles this automatically.

The Simple Approach

Running a workflow on Slurm requires just two things:

Define your jobs with resource requirements
Submit with submit-slurm

That’s it. Torc will analyze your workflow, generate appropriate Slurm configurations, and submit everything for execution.

⚠️ Important: The submit-slurm command uses heuristics to auto-generate Slurm schedulers and workflow actions. For complex workflows with unusual dependency patterns, the generated configuration may not be optimal and could result in suboptimal allocation timing. Always preview the configuration first using torc slurm generate (see Previewing Generated Configuration) before submitting production workflows.

Example Workflow

Here’s a complete workflow specification that runs on Slurm:

name: data_analysis_pipeline
description: Analyze experimental data with preprocessing, training, and evaluation

resource_requirements:
  - name: light
    num_cpus: 4
    memory: 8g
    runtime: PT30M

  - name: compute
    num_cpus: 32
    memory: 64g
    runtime: PT2H

  - name: gpu
    num_cpus: 16
    num_gpus: 2
    memory: 128g
    runtime: PT4H

jobs:
  - name: preprocess
    command: python preprocess.py --input data/ --output processed/
    resource_requirements: light

  - name: train_model
    command: python train.py --data processed/ --output model/
    resource_requirements: gpu
    depends_on: [preprocess]

  - name: evaluate
    command: python evaluate.py --model model/ --output results/
    resource_requirements: compute
    depends_on: [train_model]

  - name: generate_report
    command: python report.py --results results/
    resource_requirements: light
    depends_on: [evaluate]

Submitting the Workflow

torc submit-slurm --account myproject workflow.yaml

Torc will:

Detect which HPC system you’re on (e.g., NREL Kestrel)
Match each job’s requirements to appropriate partitions
Generate Slurm scheduler configurations
Create workflow actions that stage resource allocation based on dependencies
Submit the workflow for execution

How It Works

When you use submit-slurm, Torc performs intelligent analysis of your workflow:

1. Per-Job Scheduler Generation

Each job gets its own Slurm scheduler configuration based on its resource requirements. This means:

Jobs are matched to the most appropriate partition
Memory, CPU, and GPU requirements are correctly specified
Walltime is set to the partition’s maximum (explained below)

2. Staged Resource Allocation

Torc analyzes job dependencies and creates staged workflow actions:

Jobs without dependencies trigger on_workflow_start — resources are allocated immediately
Jobs with dependencies trigger on_jobs_ready — resources are allocated only when the job becomes ready to run

This prevents wasting allocation time on resources that aren’t needed yet. For example, in the workflow above:

preprocess resources are allocated at workflow start
train_model resources are allocated when preprocess completes
evaluate resources are allocated when train_model completes
generate_report resources are allocated when evaluate completes

3. Conservative Walltime

Torc sets the walltime to the partition’s maximum rather than your job’s estimated runtime. This provides:

Headroom for jobs that run slightly longer than expected
No additional cost since Torc workers exit when work completes
Protection against job termination due to tight time limits

For example, if your job requests 3 hours and matches the “short” partition (4 hours max), the allocation will request 4 hours.

4. HPC Profile Knowledge

Torc includes built-in knowledge of HPC systems like NREL Kestrel, including:

Available partitions and their resource limits
GPU configurations
Memory and CPU specifications
Special requirements (e.g., minimum node counts for high-bandwidth partitions)

Using an unsupported HPC? Please request built-in support so everyone benefits. You can also create a custom profile for immediate use.

Resource Requirements Specification

Resource requirements are the key to the simplified workflow. Define them once and reference them from jobs:

resource_requirements:
  - name: small
    num_cpus: 4
    num_gpus: 0
    num_nodes: 1
    memory: 8g
    runtime: PT1H

  - name: gpu_training
    num_cpus: 32
    num_gpus: 4
    num_nodes: 1
    memory: 256g
    runtime: PT8H

Fields

Field	Description	Example
`name`	Reference name for jobs	`"compute"`
`num_cpus`	CPU cores required	`32`
`num_gpus`	GPUs required (0 if none)	`2`
`num_nodes`	Nodes required	`1`
`memory`	Memory with unit suffix	`"64g"`, `"512m"`
`runtime`	ISO8601 duration	`"PT2H"`, `"PT30M"`

Runtime Format

Use ISO8601 duration format:

PT30M — 30 minutes
PT2H — 2 hours
PT1H30M — 1 hour 30 minutes
P1D — 1 day
P2DT4H — 2 days 4 hours

Job Dependencies

Define dependencies explicitly or implicitly through file/data relationships:

Explicit Dependencies

jobs:
  - name: step1
    command: ./step1.sh
    resource_requirements: small

  - name: step2
    command: ./step2.sh
    resource_requirements: small
    depends_on: [step1]

  - name: step3
    command: ./step3.sh
    resource_requirements: small
    depends_on: [step1, step2]  # Waits for both

Implicit Dependencies (via Files)

files:
  - name: raw_data
    path: /data/raw.csv
  - name: processed_data
    path: /data/processed.csv

jobs:
  - name: process
    command: python process.py
    input_files: [raw_data]
    output_files: [processed_data]
    resource_requirements: compute

  - name: analyze
    command: python analyze.py
    input_files: [processed_data]  # Creates implicit dependency on 'process'
    resource_requirements: compute

Previewing Generated Configuration

Recommended Practice: Always preview the generated configuration before submitting to Slurm, especially for complex workflows. This allows you to verify that schedulers and actions are appropriate for your workflow structure.

Viewing the Execution Plan

Before generating schedulers, visualize how your workflow will execute in stages:

torc workflows execution-plan workflow.yaml

This shows the execution stages, which jobs run at each stage, and (if schedulers are defined) when Slurm allocations are requested. See Visualizing Workflow Structure for detailed examples.

Generating Slurm Configuration

Preview what Torc will generate:

torc slurm generate --account myproject --profile kestrel workflow.yaml

This outputs the complete workflow with generated schedulers and actions:

name: data_analysis_pipeline
# ... original content ...

jobs:
  - name: preprocess
    command: python preprocess.py --input data/ --output processed/
    resource_requirements: light
    scheduler: preprocess_scheduler

  # ... more jobs ...

slurm_schedulers:
  - name: preprocess_scheduler
    account: myproject
    mem: 8g
    nodes: 1
    walltime: "04:00:00"

  - name: train_model_scheduler
    account: myproject
    mem: 128g
    nodes: 1
    gres: "gpu:2"
    walltime: "04:00:00"

  # ... more schedulers ...

actions:
  - trigger_type: on_workflow_start
    action_type: schedule_nodes
    scheduler: preprocess_scheduler
    scheduler_type: slurm
    num_allocations: 1

  - trigger_type: on_jobs_ready
    action_type: schedule_nodes
    jobs: [train_model]
    scheduler: train_model_scheduler
    scheduler_type: slurm
    num_allocations: 1

  # ... more actions ...

Save the output to inspect or modify before submission:

torc slurm generate --account myproject workflow.yaml -o workflow_with_schedulers.yaml

Torc Server Considerations

The Torc server must be accessible to compute nodes. Options include:

Shared server (Recommended): A team member allocates a dedicated server in the HPC environment
Login node: Suitable for small workflows with few, long-running jobs

For large workflows with many short jobs, a dedicated server prevents overloading login nodes.

Best Practices

1. Focus on Resource Requirements

Spend time accurately defining resource requirements. Torc handles the rest:

resource_requirements:
  # Be specific about what each job type needs
  - name: io_heavy
    num_cpus: 4
    memory: 32g      # High memory for data loading
    runtime: PT1H

  - name: compute_heavy
    num_cpus: 64
    memory: 16g      # Less memory, more CPU
    runtime: PT4H

2. Use Meaningful Names

Name resource requirements by their purpose, not by partition:

# Good - describes the workload
resource_requirements:
  - name: data_preprocessing
  - name: model_training
  - name: inference

# Avoid - ties you to specific infrastructure
resource_requirements:
  - name: short_partition
  - name: gpu_h100

3. Group Similar Jobs

Jobs with similar requirements can share resource requirement definitions:

resource_requirements:
  - name: quick_task
    num_cpus: 2
    memory: 4g
    runtime: PT15M

jobs:
  - name: validate_input
    command: ./validate.sh
    resource_requirements: quick_task

  - name: check_output
    command: ./check.sh
    resource_requirements: quick_task
    depends_on: [main_process]

4. Test Locally First

Validate your workflow logic locally before submitting to HPC:

# Run locally (without Slurm)
torc run workflow.yaml

# Then submit to HPC
torc submit-slurm --account myproject workflow.yaml

Limitations and Caveats

The auto-generation in submit-slurm uses heuristics that work well for common workflow patterns but may not be optimal for all cases:

When Auto-Generation Works Well

Linear pipelines: A → B → C → D
Fan-out patterns: One job unblocks many (e.g., preprocess → 100 work jobs)
Fan-in patterns: Many jobs unblock one (e.g., 100 work jobs → postprocess)
Simple DAGs: Clear dependency structures with distinct resource tiers

When to Use Manual Configuration

Consider using torc slurm generate to preview and manually adjust, or define schedulers manually, when:

Complex dependency graphs: Multiple interleaved dependency patterns
Shared schedulers: You want multiple jobs to share the same Slurm allocation
Custom timing: Specific requirements for when allocations should be requested
Resource optimization: Fine-tuning to minimize allocation waste
Multi-node jobs: Jobs requiring coordination across multiple nodes

What Could Go Wrong

Without previewing, auto-generation might:

Request allocations too early: Wasting queue time waiting for dependencies
Request allocations too late: Adding latency to job startup
Create suboptimal scheduler groupings: Not sharing allocations when beneficial
Miss optimization opportunities: Not recognizing patterns that could share resources

Best Practice: For production workflows, always run torc slurm generate first, review the output, and submit the reviewed configuration with torc submit.

Advanced: Manual Scheduler Configuration

For advanced users who need fine-grained control, you can define schedulers and actions manually. See Working with Slurm for details.

Common reasons for manual configuration:

Non-standard partition requirements
Custom Slurm directives (e.g., --constraint)
Multi-node jobs with specific topology requirements
Reusing allocations across multiple jobs for efficiency

Troubleshooting

“No partition found for job”

Your resource requirements exceed what’s available. Check:

Memory doesn’t exceed partition limits
Runtime doesn’t exceed partition walltime
GPU count is available on GPU partitions

Use torc hpc partitions <profile> to see available resources.

Jobs Not Starting

Ensure the Torc server is accessible from compute nodes:

# From a compute node
curl $TORC_API_URL/health

Wrong Partition Selected

Use torc hpc match to see which partitions match your requirements:

torc hpc match kestrel --cpus 32 --memory 64g --walltime 2h --gpus 2

Torc Documentation