Runtime execution#

The Flow runtime executes a compiled execution plan.

By the time runtime execution begins, the workflow has already been resolved and planned. The runtime does not need to interpret workflow.yaml, infer field dependencies, or determine which implementation an operation refers to.

Instead, it receives an explicit plan describing:

  • which nodes exist

  • how those nodes are connected

  • which implementations they use

  • which input partitions are available

  • which products they consume and produce

  • at which scope they operate

  • which execution backend should be used

        flowchart LR
    Plan["<b>Execution plan</b>"]:::plan
    Runtime["<b>Flow runtime</b><br/>execution semantics"]:::runtime
    Backend["<b>Backend</b><br/>map work to resources"]:::capability
    Implementations["<b>Registered implementations</b>"]:::capability
    Artifacts["<b>Products + artifacts</b>"]:::artifact

    Plan --> Runtime --> Backend --> Implementations --> Artifacts

classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef observer fill:#f3f0ff,stroke:#7048e8,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef plan fill:#d9ead3,stroke:#4f7d45,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;

Flow is responsible for orchestration. Registered implementations remain responsible for the actual work performed on the data.

From plan to execution#

Consider the filtering node from the exoplanet workflow:

- id: stage.EarthSizedPlanets
  role: transform
  impl: workshop.tabular.filter

  inputs:
    - node_id: stage.PlanetRows
      output_name: stream
      input_name: stream

  params:
    expr: (planet_radius > 0.8) & (planet_radius < 1.2)

  outputs:
    stream: event_stream

  input_scope: partition
  output_scope: partition

  partitioning:
    mode: dataset

  materialize: never

The runtime does not need to understand what an Earth-sized planet is or how the expression should be evaluated.

Its responsibilities are instead approximately:

  1. determine when the node is ready to run

  2. obtain the required stream product

  3. resolve the registered implementation for workshop.tabular.filter

  4. invoke its registered implementation with the planned parameters and runtime context

  5. receive the resulting event_stream

  6. make that product available to downstream nodes

The operation implementation owns the domain-specific computation.

This separation is central to Flow:

Flow
    orchestrates computation

registered implementations
    perform computation

The backend executes planned work#

Flow separates runtime orchestration from the mechanism used to execute work.

The plan selects an execution configuration:

execution:
  backend: local
  strategy: default

The resolved registry provides the available backend implementation:

backends:
  local.default:
    impl: hepflow.backends:Local

The runtime determines what work is ready to execute according to Flow’s execution semantics. The backend maps that work onto the available computing resources.

This allows Flow to support different execution systems without embedding their scheduling mechanisms into workflow operations.

Conceptually:

        flowchart TD
    Runtime["<b>Flow runtime</b>"]:::runtime

    Local["<b>Local backend</b>"]:::capability
    Distributed["<b>Distributed backend</b>"]:::capability
    Future["<b>Other backends</b>"]:::capability

    Runtime --> Local
    Runtime --> Distributed
    Runtime --> Future

classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef observer fill:#f3f0ff,stroke:#7048e8,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef plan fill:#d9ead3,stroke:#4f7d45,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;

The backend changes how work is executed, not what the individual workflow operations mean.

Execution environments and backend configuration are discussed in Execution environments.

Partition-level execution#

Many data-processing operations execute independently for each input partition.

For example, suppose a dataset has three partitions:

partition 1
partition 2
partition 3

A chain of partition-scoped transforms can be evaluated independently for each:

        flowchart LR
    P1["<b>Partition 1</b>"]:::input --> A1["PlanetRows"]:::transform --> B1["EarthSizedPlanets"]:::transform --> C1["PlanetTable"]:::transform
    P2["<b>Partition 2</b>"]:::input --> A2["PlanetRows"]:::transform --> B2["EarthSizedPlanets"]:::transform --> C2["PlanetTable"]:::transform
    P3["<b>Partition 3</b>"]:::input --> A3["PlanetRows"]:::transform --> B3["EarthSizedPlanets"]:::transform --> C3["PlanetTable"]:::transform

classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef observer fill:#f3f0ff,stroke:#7048e8,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef plan fill:#d9ead3,stroke:#4f7d45,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;

Each execution follows the same nodes and parameters, but operates on a different partition of the input data.

This is one of the main sources of parallelism available to execution backends.

Flow describes these operations in terms of products and partitions rather than requiring implementations to manage distributed scheduling themselves.

Partition-boundary products#

Partition execution has an explicit boundary. A product may be useful inside one partition but too large or too short-lived to return from that partition to the driver.

Flow separates two questions:

  • Liveness: does this product need to survive the partition?

  • Representation: if it survives, what form crosses the boundary?

Liveness is determined from the execution plan and product-handler policy. A product crosses the partition boundary when either a wider-scope consumer needs it, such as a dataset-level or run-level sink, or its registered product handler declares that it should be retained even without a wider graph consumer.

Representation is declared independently by the product handler:

  • value carries the runtime value itself, suitable for small mergeable products such as histograms, cutflows, or counters.

  • reference carries an already durable/lightweight reference, such as an output artifact result describing a file that was written during partition execution.

  • materialize reserves the lifecycle where an in-memory value must be converted into a durable/lightweight boundary representation before it leaves the partition.

The two invariants are:

  • A partition executor must return only products whose lifetime extends beyond the partition.

  • A product crossing a partition boundary must cross in the cheapest representation appropriate to its product type.

This first boundary model lets Flow plan and extract BoundaryProduct wrappers without making generic runtime code aware of ROOT, Awkward, histograms, Carpenter, or any other package-specific value type.

Backend adoption, local executor retention changes, and incremental reduction are later phases. The current runtime may still retain complete partition value stores while the boundary contract is being introduced.

Products carry results between nodes#

Operations communicate through products.

For the exoplanet workflow:

read.planets
    │
    │ event_stream
    ▼
PlanetRows
    │
    │ event_stream
    ▼
EarthSizedPlanets
    │
    │ event_stream
    ▼
PlanetTable
    │
    │ event_stream
    ▼
console_table

Each node consumes named inputs and produces named outputs.

For example:

inputs:
  - node_id: stage.PlanetRows
    output_name: stream
    input_name: stream

outputs:
  stream: event_stream

The runtime uses these relationships to route products between implementations.

It does not need to know the internal representation of every product type.

Product types can define runtime behaviour#

Some products require behaviour beyond simply passing a Python value from one operation to another.

For example, partition-level results may need to be combined before they can be consumed by a global operation.

Flow supports registered product handlers for this purpose.

A registry may contain entries such as:

product_handlers:
  event_stream:
    merge: ...

  histogram:
    merge: ...
    materialize: ...

Product handlers allow packages that define product types to also define how those products behave at runtime.

Depending on the product, this may include:

  • merging partition results

  • converting a runtime product into a persistent artifact

  • preparing a product for another execution scope

This follows the same extensibility principle as operations: Flow provides the orchestration mechanism, while packages that understand a product provide its specialised behaviour.

Moving between scopes#

Not every operation runs at partition scope.

The plan distinguishes between:

partition
dataset
global

A partition-scoped operation can execute independently for each partition.

A dataset-scoped operation consumes information associated with a complete dataset.

A global operation consumes information across the complete workflow.

Consider the final table writer from the exoplanet example:

input_scope: global
output_scope: global

while the preceding transform produces:

output_scope: partition

The runtime therefore needs to cross a scope boundary:

        flowchart TD
    P1["<b>Partition product</b>"]:::artifact
    P2["<b>Partition product</b>"]:::artifact
    P3["<b>Partition product</b>"]:::artifact

    Merge["<b>Product handling</b><br/>merge / promote"]:::capability
    Global["<b>Global product</b>"]:::artifact
    Sink["<b>Global sink</b>"]:::sink

    P1 --> Merge
    P2 --> Merge
    P3 --> Merge
    Merge --> Global --> Sink

classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef observer fill:#f3f0ff,stroke:#7048e8,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef plan fill:#d9ead3,stroke:#4f7d45,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;

How products are combined depends on their product type.

A histogram, event stream, cutflow, or custom analysis product may require different merge semantics.

Those semantics belong to the corresponding product handler rather than being hardcoded into Flow.

Materialisation#

Runtime products do not necessarily become persistent artifacts.

Intermediate products may exist only long enough to be consumed by downstream operations:

materialize: never

Other nodes explicitly produce outputs that should survive execution:

materialize: always

Materialisation may involve writing:

  • files

  • histograms

  • tables

  • reports

  • other persistent artifacts

The details depend on the product and its registered capabilities.

This distinction allows Flow to reason separately about:

a value needed during execution

and:

an artifact that should remain after execution

Sources participate in the same model#

Sources are runtime implementations too.

For the exoplanet workflow, the source node is:

- id: read.planets
  role: source
  impl: workshop.parquet

  params:
    stream_type: event_stream
    branches:
      - name
      - planet_name
      - planet_period
      - planet_radius

  outputs:
    stream: event_stream

The compiler has already determined which fields are required.

At runtime, the source implementation receives this resolved request together with information about the partition being processed.

The source then decides how to satisfy that request.

For a Parquet source this might mean column projection. A ROOT source may interpret the same idea as branch selection. Another source may obtain its data from an entirely different storage system.

Flow does not need to understand those storage details.

Sinks complete data-flow paths#

Sinks consume products and produce final outputs or artifacts.

The exoplanet workflow ends with:

- id: write.PlanetTable.0
  role: sink
  impl: workshop.console_table

  inputs:
    - node_id: stage.PlanetTable
      output_name: stream
      input_name: target

  outputs:
    artifact: artifact

  input_scope: global
  output_scope: global

  materialize: always

The sink is responsible for turning the input product into the requested output.

In this example, that means producing the final text table:

+---------------+----------------------+---------------+
| Planet        | Radius [Earth radii] | Period [days] |
+---------------+----------------------+---------------+
| Kepler-950 b  | 0.801                | 98.718        |
| Kepler-51 c   | 0.803                | 85.312        |
...
+---------------+----------------------+---------------+

Flow determines when the sink can execute and supplies its input. The sink implementation determines how the table is produced.

Runtime context#

Implementations often need more than their immediate input product and parameters.

The runtime provides context describing the execution in which an operation is running.

Depending on the operation and execution scope, this can include information about:

  • the current dataset

  • the current partition

  • workflow-wide context

  • artifact locations

  • execution configuration

  • registered capabilities

  • runtime services

This avoids requiring operations to discover workflow state through global variables or hidden configuration.

The precise runtime context contract is part of the extension API and is documented alongside custom operation development.

Runtime extensions#

Not all runtime behaviour belongs in the logical data-flow structure of the workflow.

Execution hooks allow extensions to react to runtime lifecycle events.

These can support concerns such as:

  • diagnostics

  • metrics

  • logging

  • provenance

  • execution summaries

  • resource monitoring

Conceptually:

run starts
    │
    ├── execution hooks
    │
    ▼
partitions and nodes execute
    │
    ├── execution hooks
    │
    ▼
aggregation / finalisation
    │
    ├── execution hooks
    │
    ▼
run ends

This keeps cross-cutting runtime behaviour separate from scientific data-processing operations.

The set of hooks is resolved during compilation and recorded in the plan.

Execution modifiers#

Some execution environments require behaviour that affects how operations are executed without changing the operation itself.

Flow supports registered execution modifiers for this purpose.

Examples may include:

  • preparing accelerator state

  • loading runtime libraries

  • compiling specialised kernels

  • modifying execution context

  • backend-specific setup

For example, an extension might provide capabilities such as:

execution_modifiers:
  gpu.preload:
    impl: ...

  cuda.jit:
    impl: ...

The important distinction is that these modify execution behaviour, not workflow semantics.

The scientific workflow should not need to change merely because an implementation requires different hardware or runtime preparation.

Runtime diagnostics and reports#

Runtime execution can produce more than the scientific outputs of the workflow.

Depending on the active profiles and hooks, a run may also generate:

debug/
├── dask/
├── logs/
└── performance/

reports/
├── diagnostics/
├── provenance/
└── schema/

run_summary.yaml

These outputs can describe:

  • what was executed

  • which backend was used

  • runtime failures or warnings

  • performance information

  • provenance

  • generated artifacts

Such capabilities are generally provided through extensions rather than being intrinsic to every Flow execution.

This keeps the runtime small while allowing richer environments to add inspection and diagnostics when required.

Resource Provenance#

Runtime provenance records the logical resource identity used by an operation and, for file-backed resources that provide it, the resolved content identity.

For example, a workflow can ask for a CMS-recommended logical payload such as a Winter24 Jet ID resource. At runtime that logical path may currently resolve through a filesystem symlink to a particular payload file. The provenance record keeps both pieces of information:

  • the FAST-HEP resource id and configured/logical path;

  • the resolved filesystem path actually opened;

  • the selected correction object and correction version when available;

  • the payload size and SHA-256 of the exact bytes consumed.

If the upstream recommendation later retargets the same logical path to new bytes, the workflow can remain unchanged while the runtime provenance changes. Productions can then be compared by their recorded payload fingerprint rather than by relying on the current state of an external filesystem.

Flow does not perform the scientific computation#

The runtime deliberately has a narrow responsibility.

For a transform such as:

workshop.tabular.filter

Flow does not implement filtering.

For a source such as:

workshop.parquet

Flow does not implement Parquet reading.

For a HEP operation such as:

hep.hist

Flow does not implement histogramming.

Instead:

        flowchart TD
    Flow["<b>Flow runtime</b><br/>orchestration"]:::runtime

    Source["<b>Source implementation</b>"]:::source
    Transform["<b>Transform implementation</b>"]:::transform
    Sink["<b>Sink implementation</b>"]:::sink
    Handler["<b>Product handler</b>"]:::capability

    Flow --> Source
    Flow --> Transform
    Flow --> Sink
    Flow --> Handler

classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef observer fill:#f3f0ff,stroke:#7048e8,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef plan fill:#d9ead3,stroke:#4f7d45,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;

This separation allows implementations to evolve independently of the workflow engine.

A capability can be replaced with a new implementation — using a different library, data representation, algorithm, or computing architecture — while retaining the orchestration model and, where contracts remain compatible, the workflow itself.

Runtime versus backend#

The terms runtime and backend describe different layers.

The runtime understands Flow execution semantics:

  • nodes and dependencies

  • products and scopes

  • partitions

  • materialisation

  • lifecycle hooks

The backend understands how executable work is mapped onto resources:

  • directly in the current process

  • on local workers

  • through a distributed scheduler

  • in another execution environment

A useful approximation is:

        flowchart LR
    Plan["<b>Execution plan</b>"]:::plan
    Runtime["<b>Flow runtime</b><br/>what is ready to run?"]:::runtime
    Backend["<b>Backend</b><br/>where / how does it run?"]:::capability
    Impl["<b>Implementation</b><br/>perform the work"]:::transform

    Plan --> Runtime --> Backend --> Impl

classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef observer fill:#f3f0ff,stroke:#7048e8,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef plan fill:#d9ead3,stroke:#4f7d45,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;

The boundary allows execution infrastructure to evolve without becoming part of operation semantics.

Execution produces artifacts and a run record#

When execution completes, scientific outputs are written beneath the artifact directory.

For the exoplanet example:

artifacts/
└── files/
    └── snippets/
        └── planets.txt

The run also produces:

run_summary.yaml

The summary provides a top-level record of the execution and generated outputs.

Additional artifacts, reports, and diagnostics depend on the capabilities activated for the workflow.

The runtime starts from the plan#

The most important runtime boundary is therefore:

        flowchart TD
    Plan["<b>Execution plan</b>"]:::plan
    Runtime["<b>Flow runtime</b><br/>orchestration"]:::runtime

    Backend["<b>Backend</b><br/>execution resources"]:::capability
    Operations["<b>Operations</b><br/>scientific computation"]:::capability
    Extensions["<b>Extensions</b><br/>lifecycle behaviour"]:::capability

    Outputs["<b>Products + artifacts</b>"]:::artifact

    Plan --> Runtime
    Runtime --> Backend
    Runtime --> Operations
    Runtime --> Extensions
    Operations --> Outputs

classDef input fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef flow fill:#e8f0fe,stroke:#3c6fbd,stroke-width:2px,color:#111111;
classDef source fill:#e7f5ff,stroke:#1c7ed6,stroke-width:2px,color:#111111;
classDef transform fill:#fff9db,stroke:#f08c00,stroke-width:2px,color:#111111;
classDef observer fill:#f3f0ff,stroke:#7048e8,stroke-width:2px,color:#111111;
classDef sink fill:#ebfbee,stroke:#2f9e44,stroke-width:2px,color:#111111;
classDef capability fill:#f7f7f7,stroke:#5f6368,stroke-width:2px,color:#111111;
classDef plan fill:#d9ead3,stroke:#4f7d45,stroke-width:2px,color:#111111;
classDef runtime fill:#fce8d5,stroke:#b56b22,stroke-width:2px,color:#111111;
classDef artifact fill:#eadcf8,stroke:#7950a3,stroke-width:2px,color:#111111;

The runtime does not need to know how the plan was authored.

Its job is to orchestrate the resolved computation represented by that plan.

This is what makes the execution plan a meaningful interface between compilation and execution, independent of the workflow language that produced it.

Where next?#

This page described how Flow executes a plan.

Continue with Execution environments for how execution is mapped onto local and distributed computing environments.

For the structure consumed by the runtime, see The execution plan.

For how that plan is constructed, see Workflow compilation.

For implementing capabilities that Flow can orchestrate, see Extending Flow.