Executors - run your workflows¶
fasthep-flow
is currently built on top of
prefect, which means it supports all of the
executors, or in this case “Task Runners”, that prefect supports. The default
executor is the SequentialTaskRunner, which runs all
tasks in sequence on the local machine. A full list of executors can be found in
the
prefect documentation.
Since prefect is not widely used in High Energy Particle Physics, let’s go over the executors that are most relevant to us.
SequentialTaskRunner¶
The SequentialTaskRunner
(see
prefect docs)
runs each task in a separate process on the local machine. This is the default
executor for fasthep-flow
.
DaskTaskRunner¶
The DaskTaskRunner
(see
prefect docs) runs each task in a
separate process on a Dask cluster. A Dask cluster can be run on a local machine
or as a distributed cluster using a batch system (e.g. HTCondor, LSF, PBS, SGE,
SLURM) or other distributed systems such as LHCb’s DIRAC. This is the
recommended executor for running fasthep-flow
workflows on distributed
resources.
Custom executors¶
Documentation on how to create custom executors can be found in the developer’s corner.