Running a Multistep Workflow Locally

Sophios can run Python-authored workflows locally through supported CWL runners. The default runner is cwltool. The other supported local runner is toil-cwl-runner.

This page uses a three-step workflow so the runner configuration is easy to see:

touch.cwl -> append.cwl -> cat.cwl -> workflow output "result"

A runnable version lives in examples/scripts/multistep_runner_pyapi.py.

Build the Workflow

from pathlib import Path

from sophios.apis.python.workflow import Step, Workflow


ADAPTERS = Path("cwl_adapters")

touch = Step(ADAPTERS / "touch.cwl")
touch.inputs.filename = "empty.txt"

append = Step(ADAPTERS / "append.cwl")
append.inputs.file = touch.outputs.file
append.inputs.str = "Hello"

cat = Step(ADAPTERS / "cat.cwl")
cat.inputs.file = append.outputs.file

workflow = Workflow([touch, append, cat], "multistep_runner_pyapi")
workflow.outputs.result = cat.outputs.output

The workflow code does not change when you switch local runners. Runner choice belongs to execution configuration, not to the workflow graph.

Run With the Default Runner

workflow.run()

This compiles the workflow, writes the generated CWL and job inputs, and runs the workflow locally with cwltool.

Configure cwltool

Pass local-run settings through run_args_dict:

workflow.run(
    basepath="autogenerated/multistep_cwltool",
    run_args_dict={
        "cwl_runner": "cwltool",
        "container_engine": "docker",
        "copy_output_files": "yes",
    },
)

Common Sophios local-run settings include:

• cwl_runner: either cwltool or toil-cwl-runner.

• container_engine: the container command used by the runner, usually docker or podman.

• copy_output_files: use "yes" to copy primary outputs into outdir/ after a successful local run.

• cachedir: override the cache directory used by local execution.

• generate_run_script: use "yes" to write run.sh for inspection instead of invoking the runner.

Run With Toil

To run the same workflow with Toil, change only the runner configuration:

workflow.run(
    basepath="autogenerated/multistep_toil",
    run_args_dict={
        "cwl_runner": "toil-cwl-runner",
        "container_engine": "docker",
        "logLevel": "INFO",
    },
)

Runner-specific key/value options can be included in run_args_dict. Sophios uses the options it recognizes for local setup and passes the remaining key/value options to the selected CWL runner. In the Toil example above, "logLevel": "INFO" is passed to Toil as a runner option.

Inspect the Runner Command

When you want to see the generated command without executing the runner, ask Sophios to write a run script:

workflow.run(
    run_args_dict={
        "cwl_runner": "cwltool",
        "container_engine": "docker",
        "generate_run_script": "yes",
    },
)

This still compiles the workflow and prepares local execution artifacts, but it stops before invoking the selected runner.

Run the Example Script

python examples/scripts/multistep_runner_pyapi.py

The script is intentionally a normal Python build/run script, not a separate CLI app. The runner configuration is near the top:

RUN_ARGS = {
    "cwl_runner": "cwltool",
    "container_engine": "docker",
    "copy_output_files": "yes",
}

To use Toil, change the same dictionary:

RUN_ARGS = {
    "cwl_runner": "toil-cwl-runner",
    "container_engine": "docker",
    "logLevel": "INFO",
}

That keeps the build script readable and portable as Python source while Sophios generates the portable CWL workflow for execution.