What is a DAG?
It stands for a Directed Acyclic Graph and is the basis of Airflow, Prefect, and Dagster.
Lately I’ve spent a lot of time researching different orchestration tools to use within a data stack that I’m building at. I’ve been looking at Airflow, Dagster, and Prefect to name a few.
As someone who has never used one of these tools before, it was a bit confusing reading through documentation. I mean how many times are they going to mention DAG without actually explaining what DAG means!?
As a self-taught programmer who has learned most of what she knows on the job and through various tutorials and articles online, I needed to dive deep into what these tools meant when they referred to DAG. If you’re in the same boat as I was, hopefully this helps you out.
DAG stands for Directed Acyclic Graph. This is a concept often used in mathematics and computer science. It is basically a graph with arrows pointing from one event to another, forming a cycle that never really closes.
A DAG is a way of defining relationships and dependencies between different events. It shows you the order in which they have to be executed and which events depend on one another.
Looking at this picture, events 1, 2, and 3 run in parallel and must be completed before event 4 can run. Event 4 depends on events 1, 2, and 3. After event 4 runs and completes, events 5, 6, and 7 can then be run in parallel. Lastly, when these 3 events are all done running then event 8 can be executed. Event 8 depends on the success of events 1–8 in order to be successfully executed. These dependencies are the bones of a DAG.
You can read more about DAGs in Airflow’s documentation.
Have you used any of these orchestration tools? Let me know what your experience was like!
