dag_from_scratch.rst

orphan

Scheduled DAG with pandas and sklearn from scratch.

Audience: Users coming from MLOps to Lightning Apps, looking for more flexibility.

Level: Intermediate.

In this example, you will learn how to create a simple DAG which:

• Download and process some data
• Train several models and report their associated metrics

and learn how to schedule this entire process.

Find the complete example here.

---

Step 1: Implement your DAG

Here is an illustration of the DAG to implement:

First, let's define the component we need:

• DataCollector is responsible to download the data
• Processing is responsible to execute a processing.py script.
• A collection of model work to train all models in parallel.

../../../examples/app_dag/app.py

And its run method executes the steps described above.

../../../examples/app_dag/app.py

---

Step 2: Define the scheduling

../../../examples/app_dag/app.py