Tired of Airflow? Try This.

A new tool that does what Airflow does but better

When I first joined my current company my first major project was deciding on a data stack we wanted to use for analytics. I researched the best solutions for ingestion, warehousing, transformation, and orchestration. All the others were fairly straightforward but the orchestration part stumped me.

Sure, Airflow may be great if you have a whole team dedicated to working with it. However, with a small startup, you have one engineer that works on all aspects of the data stack. And that one engineer was me.

I didn’t want to spend time monitoring and maintaining a complex Airflow data pipeline when that’s just not what we needed. We wanted something that would get the job done but with as little maintenance as possible. After all, I had other problems to solve!

That’s when I discovered a newer orchestration solution called Prefect.

Prefect’s philosophy revolves around the idea of negative engineering, meaning it is helpful when something goes wrong. It considers itself a risk management tool. You shouldn’t have to worry about it UNLESS your pipeline isn’t working as expected, then it is extremely useful.

What is Prefect?

Prefect considers itself a workflow engine. It is there to help you build, run, and monitor your data pipelines. Or, if you’re thinking in terms of a data stack, it is the very last tool in the stack- the orchestration tool.

Prefect pipelines are written in Python using tasks and flows. If you know how to write a Python function, you can easily build a data pipeline using Prefect.

Tasks are individual jobs that do one unit of work. In my data pipeline, this is usually a step that syncs Fivetran data or runs a dbt model.

Flows are functions that consist of a bunch of smaller tasks, or units of work, that depend on one another. In my case, my data pipeline is one flow consisting of tasks running Fivetran syncs and dbt models.

These flows are then scheduled and run by whatever types of agents you choose to set up. Some options include local, Docker, or Kubernetes. Agents will pick up the flows and run them according to the schedule you put in place.

Why Prefect over Airflow?

Personally, I just thought Prefect was a lot easier to use and way more sophisticated in terms of its UI. Is it just me or is Airflow ugly and just not that useful in telling your pipeline’s story?

As someone with a background in data engineering but no experience using Airflow, I was pretty confused trying to set it up. I wanted to set it up using Astronomer but even that was just complicated. I had way too many screens popping up every time I tried to launch the application and none of them gave me the information I needed.

Not to mention the support team at Astronomer was almost inexistent whereas Prefect has a Slack community where you can get all your questions answered within minutes. The Slack community is one of my favorite parts about dbt so this was a huge plus for me.

Overall, the time to set up Airflow vs Prefect was a huge factor for me. As a scrappy start-up, I wanted to hit the ground running with improving our data models. I didn’t want to spend months just getting Airflow to work. Prefect took me around two weeks to get working in its entirety and I haven’t had any problems since.

Another huge factor in my decision was the UI. With Airflow I had to really dig for the answer to my questions on why something went wrong. The UI was hard to read and didn’t communicate failures/successes very well. I mean, take a look for yourself.

Is this straightforward to you? To me, it wasn’t. How do I know which task failed and when? Tracing back the little red failure boxes wasn’t so easy.

Now look at Prefect’s UI. You can easily see each task within the larger flow and whether it succeed, failed, or is still in the queue. I can easily scan this, click on the take that failed, read the logs, and re-run it. Something that would take me minutes in Prefect may take me an hour in Airflow. And an hour is valuable time.

Conclusion

If you’re deciding on an orchestration solution for your analytics data stack, I highly recommended considering Prefect. While everyone’s use case vary, Prefect is the perfect autonomous scheduler for an analytics engineer who wants as little maintenance as possible.

I simply set up a Slack webhook to notify me whenever a task fails and that is the only time I ever need to check on things. I know if I don’t receive a failure message, my data is dependable. And that is a great feeling! One less thing to worry about when starting the work day.

Let me know what you think and if you’d be interested in hearing more about how I’m using Prefect and how I set it up. The documentation can be a bit confusing at times but it was still infinitely easier for me than working with Airflow. And the low maintenance and reliability have been absolutely worth the set-up.

Learn more about other popular modern data stack tools by subscribing to my email list.