Creating Workflows with the Dashboard Wizard

This tutorial walks you through creating a workflow using the interactive wizard in the Torc dashboard. The wizard provides a guided, step-by-step interface for building workflows without writing YAML or JSON files.

Learning Objectives

By the end of this tutorial, you will:

Create a multi-job workflow using the dashboard wizard
Define job dependencies visually
Configure Slurm schedulers for HPC execution
Set up workflow actions to automatically schedule nodes
Understand how the wizard generates workflow specifications

Prerequisites

Torc dashboard running (see Dashboard Deployment)
Basic understanding of Torc workflows

Overview

The workflow wizard guides you through five steps:

Basics - Workflow name and description
Jobs - Define computational tasks
Schedulers - Configure Slurm schedulers (optional)
Actions - Set up automatic node scheduling (optional)
Review - Preview and create the workflow

Open the Torc dashboard in your browser
Click the Create Workflow button in the top-right corner
Select the Wizard tab at the top of the modal

You’ll see the wizard interface with step indicators showing your progress.

Step 2: Configure Basics

Enter the basic workflow information:

Workflow Name (required): A unique identifier for your workflow (e.g., data-pipeline)
Description (optional): A brief description of what the workflow does

Click Next to proceed.

Step 3: Add Jobs

This is where you define the computational tasks in your workflow.

Adding Your First Job

Click + Add Job
Fill in the job details:
- Job Name: A unique name (e.g., preprocess)
- Command: The shell command to execute (e.g., python preprocess.py)

Setting Dependencies

The Blocked By field lets you specify which jobs must complete before this job can run:

Click the Blocked By dropdown
Select one or more jobs that must complete first
Hold Ctrl/Cmd to select multiple jobs

Configuring Resources

Choose a resource preset or customize:

Small: 1 CPU, 1GB memory
Medium: 8 CPUs, 50GB memory
GPU: 1 CPU, 10GB memory, 1 GPU
Custom: Specify exact requirements

Example: Three-Job Pipeline

Let’s create a simple pipeline:

Job 1: preprocess

Name: preprocess
Command: echo "Preprocessing..." && sleep 5
Blocked By: (none - this runs first)
Resources: Small

Job 2: analyze

Name: analyze
Command: echo "Analyzing..." && sleep 10
Blocked By: preprocess
Resources: Medium

Job 3: report

Name: report
Command: echo "Generating report..." && sleep 3
Blocked By: analyze
Resources: Small

Click Next when all jobs are configured.

Step 4: Configure Schedulers (Optional)

If you’re running on an HPC system with Slurm, you can define scheduler configurations here. Skip this step for local execution.

Adding a Scheduler

Click + Add Scheduler
Fill in the required fields:
- Scheduler Name: A reference name (e.g., compute_scheduler)
- Account: Your Slurm account name
Configure optional settings:
- Nodes: Number of nodes to request
- Wall Time: Maximum runtime (HH:MM:SS format)
- Partition: Slurm partition name
- QoS: Quality of service level
- GRES: GPU resources (e.g., gpu:2)
- Memory: Memory per node (e.g., 64G)
- Temp Storage: Local scratch space
- Extra Slurm Options: Additional sbatch flags

Example: Basic Compute Scheduler

Scheduler Name: compute
Account: my_project
Nodes: 1
Wall Time: 02:00:00
Partition: standard

Assigning Jobs to Schedulers

After defining schedulers, you can assign jobs to them:

Go back to the Jobs step (click Back)
In each job card, find the Scheduler dropdown
Select the scheduler to use for that job

Jobs without a scheduler assigned will run locally.

Click Next when scheduler configuration is complete.

Step 5: Configure Actions (Optional)

Actions automatically schedule Slurm nodes when certain events occur. This is useful for dynamic resource allocation.

Trigger Types

When workflow starts: Schedule nodes immediately when the workflow begins
When jobs become ready: Schedule nodes when specific jobs are ready to run
When jobs complete: Schedule nodes after specific jobs finish

Adding an Action

Click + Add Action
Select the Trigger type
Select the Scheduler to use
For job-based triggers, select which Jobs trigger the action
Set the Number of Allocations (how many Slurm jobs to submit)

Example: Stage-Based Scheduling

For a workflow with setup, compute, and finalize stages:

Action 1: Setup Stage

Trigger: When workflow starts
Scheduler: setup_scheduler
Allocations: 1

Action 2: Compute Stage

Trigger: When jobs become ready
Jobs: compute_job1, compute_job2, compute_job3
Scheduler: compute_scheduler
Allocations: 3

Action 3: Finalize Stage

Trigger: When jobs become ready
Jobs: finalize
Scheduler: finalize_scheduler
Allocations: 1

Click Next to proceed to review.

Step 6: Review and Create

The review step shows the generated workflow specification in JSON format. This is exactly what will be submitted to the server.

Reviewing the Spec

Examine the generated specification to verify:

All jobs are included with correct names and commands
Dependencies (depends_on) match your intended workflow structure
Resource requirements are correctly assigned
Schedulers have the right configuration
Actions trigger on the expected events

Creating the Workflow

Review the Options below the wizard:
- Initialize workflow after creation: Builds the dependency graph (recommended)
- Run workflow immediately: Starts execution right away
Click Create to submit the workflow

If successful, you’ll see a success notification and the workflow will appear in your workflow list.

Example: Complete Diamond Workflow

Here’s how to create a diamond-pattern workflow using the wizard:

     preprocess
       /    \
    work1   work2
       \    /
    postprocess

Jobs Configuration

Job	Command	Blocked By	Resources
preprocess	`./preprocess.sh`	(none)	Small
work1	`./work1.sh`	preprocess	Medium
work2	`./work2.sh`	preprocess	Medium
postprocess	`./postprocess.sh`	work1, work2	Small

Generated Spec Preview

The wizard generates a spec like this:

{
  "name": "diamond-workflow",
  "description": "Fan-out and fan-in example",
  "jobs": [
    {
      "name": "preprocess",
      "command": "./preprocess.sh",
      "resource_requirements": "res_1cpu_1g"
    },
    {
      "name": "work1",
      "command": "./work1.sh",
      "depends_on": ["preprocess"],
      "resource_requirements": "res_8cpu_50g"
    },
    {
      "name": "work2",
      "command": "./work2.sh",
      "depends_on": ["preprocess"],
      "resource_requirements": "res_8cpu_50g"
    },
    {
      "name": "postprocess",
      "command": "./postprocess.sh",
      "depends_on": ["work1", "work2"],
      "resource_requirements": "res_1cpu_1g"
    }
  ],
  "resource_requirements": [
    {"name": "res_1cpu_1g", "num_cpus": 1, "memory": "1g", "num_gpus": 0, "num_nodes": 1, "runtime": "PT1H"},
    {"name": "res_8cpu_50g", "num_cpus": 8, "memory": "50g", "num_gpus": 0, "num_nodes": 1, "runtime": "PT1H"}
  ]
}