Hello world¶
Sometimes you just want to see some code. This section contains some real-life
examples of how to use fasthep-flow
.
stages:
- name: "hello_world in bash"
type: "fasthep_flow.operators.BashOperator"
kwargs:
bash_command: echo "Hello World!"
Save this to a file called hello_world.yaml
.
fasthep-flow execute hello_world.yaml
This will print “Hello World!” to the console.
So far so good, but what does it actually do? Let’s to execute this step-by-step.
Creating a flow¶
The first thing that fasthep-flow
does is to create a flow. This is done by
creating a prefect.Flow
object, and adding a task for each step in the YAML
file. The task is created by the fasthep-flow
operator, and the parameters are
passed to the task as keyword arguments.
We can do this ourselves by creating a flow and adding a task to it.
from fasthep.operators import BashOperator
from prefect import Flow
flow = Flow("hello_world")
task = BashOperator(bash_command="echo 'Hello World!'")
flow.add_task(task)
Running the flow¶
Next we have to decide how to execute this flow. By default, fasthep-flow
will
run the flow on the local machine. This is done by calling flow.run()
.
flow.run()
Running the flow on a cluster¶
The real strength of fasthep-flow
is that it can run the flow on a cluster
with the same config file. Internally, this is done by creating a Dask workflow
first, and then running it on the specified cluster (e.g. HTCondor or Google
Cloud Composer). For now, let’s just run it on a local Dask cluster.
fasthep-flow execute hello_world.yaml --executor DaskLocal
This will start a Dask cluster on your local machine, and run the flow on it. While the output will be the same, you will find additional output files for Dask performance.
Provenance¶
In a real-world scenario, you would want to keep track of the provenance of your
flow. This is done automatically by fasthep-flow
, and you can find the
provenance in the output/provenance
folder.
For more information, see Provenance.
So what does this look like for our hello world example?
tree output
Next steps¶
This was a very simple example, but it shows the basic concepts of
fasthep-flow
. For more realistic examples, see the experiment specific
examples in Examples. For more advanced examples, see
Advanced Examples.