Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Tutorial 1: Many Independent Jobs

This tutorial teaches you how to create and run a workflow with many independent parallel jobs using Torc’s parameterization feature.

Learning Objectives

By the end of this tutorial, you will:

  • Understand how to define parameterized jobs that expand into multiple instances
  • Learn how Torc executes independent jobs in parallel
  • Know how to monitor job execution and view results

Prerequisites

  • Torc server running (see Installation)
  • Basic familiarity with YAML syntax

Use Cases

This pattern is ideal for:

  • Parameter sweeps: Testing different configurations
  • Monte Carlo simulations: Running many independent trials
  • Batch processing: Processing many files with the same logic
  • Embarrassingly parallel workloads: Any task that can be split into independent units

Step 1: Start the Torc Server

First, ensure the Torc server is running:

torc-server run

By default, the server listens on port 8080, making the API URL http://localhost:8080/torc-service/v1.

If you use a custom port, set the environment variable:

export TORC_API_URL="http://localhost:8100/torc-service/v1"

Step 2: Create the Workflow Specification

Save the following as hundred_jobs.yaml:

name: hundred_jobs_parallel
description: 100 independent jobs that can run in parallel

jobs:
  - name: job_{i:03d}
    command: |
      echo "Running job {i}"
      sleep $((RANDOM % 10 + 1))
      echo "Job {i} completed"
    resource_requirements: minimal
    parameters:
      i: "1:100"

resource_requirements:
  - name: minimal
    num_cpus: 1
    num_gpus: 0
    num_nodes: 1
    memory: 1g
    runtime: PT5M

Understanding the Specification

Let’s break down the key elements:

  • name: job_{i:03d}: The {i:03d} is a parameter placeholder. The :03d format specifier means “3-digit zero-padded integer”, so jobs will be named job_001, job_002, …, job_100.

  • parameters: i: "1:100": This defines a parameter i that ranges from 1 to 100 (inclusive). Torc will create one job for each value.

  • resource_requirements: minimal: Each job uses the “minimal” resource profile defined below.

When Torc processes this specification, it expands the single job definition into 100 separate jobs, each with its own parameter value substituted.

Step 3: Run the Workflow

Create and run the workflow in one command:

torc run hundred_jobs.yaml

This command:

  1. Creates the workflow on the server
  2. Expands the parameterized job into 100 individual jobs
  3. Initializes the dependency graph (in this case, no dependencies)
  4. Starts executing jobs in parallel

You’ll see output showing the workflow ID and progress.

Step 4: Monitor Execution

While the workflow runs, you can monitor progress:

# Check workflow status
torc workflows status <workflow_id>

# List jobs and their states
torc jobs list <workflow_id>

# Or use the interactive TUI
torc tui

Since all 100 jobs are independent (no dependencies between them), Torc will run as many in parallel as your system resources allow.

Step 5: View Results

After completion, check the results:

torc results list <workflow_id>

This shows return codes, execution times, and resource usage for each job.

How It Works

When you run this workflow, Torc:

  1. Expands parameters: The single job definition becomes 100 jobs (job_001 through job_100)
  2. Marks all as ready: Since there are no dependencies, all jobs start in the “ready” state
  3. Executes in parallel: The job runner claims and executes jobs based on available resources
  4. Tracks completion: Each job’s return code and metrics are recorded

The job runner respects the resource requirements you specified. With num_cpus: 1 per job, if your machine has 8 CPUs, approximately 8 jobs will run simultaneously.

What You Learned

In this tutorial, you learned how to:

  • ✅ Use parameter expansion (parameters: i: "1:100") to generate multiple jobs from one definition
  • ✅ Use format specifiers ({i:03d}) for consistent naming
  • ✅ Run independent parallel jobs with torc run
  • ✅ Monitor workflow progress and view results

Example Files

See hundred_jobs_parameterized.yaml for a ready-to-run version of this workflow.

Next Steps