The New Buzzword in Data Engineering: Zero ETL
What is Zero ETL — Definition, Benefits & Challenges
In the field of Data Engineering, we often hear about the so-called zero ETL approach, but what exactly is this?
Definition
The Zero ETL approach is a method for building data pipelines that aims to eliminate the need for traditional extraction, transformation, and loading (ETL) processes and the tools used to perform them. This approach is based on the idea that data should be stored and processed or even just analyzed within the source system e.g. with SQL in its original format without the need for complex data transformation or movement.
Benefits
At the end of the day, it means that modern cloud-based Data Warehouses, Data Lakes or even Data Lakehouses use the integrated services of the large cloud providers to analyze data directly from other sources. So rather than filtering data from SQL or NoSQL databases, processing and then putting it into your Data Lake or Data Warehouse, etc. two times, one can just easily gain access to the data directly (often simply via SQL). This has several advantages, like:
- Less effort for building up data pipelines, especially less effort if you have previously programmed them.
- No double existing data storage, which unnecessarily take up money and cause a poorer performance.
- In some cases maybe also no expensive data integrations solution like talend, alteryx & Co.
Another main benefit of the Zero ETL approach is that it allows organizations to work with data in real time, rather than waiting for data to be extracted, transformed and loaded into a separate system.
Challenges
With all these benefits and less effort in data integration, one may naturally ask: It the Data Engineer needed no longer? Will the Data Scientist soon be able to provide their data on their own? This is exactly the question I explored in the article below.
Not to create too much suspense, a little spoiler: No Data Engineers are still needed, but their field of activity may shift. For example one of the biggest challenges of the Zero ETL approach is that it requires significant upfront planning and design. Organizations and especially the Data Engineer need to consider their data architecture, processing requirements and scalability before implementing a Zero-ETL pipeline. Also, the subsequent processes still need data transformation and aggregation logics. If data is analyzed directly from sources or loaded untransformed, for example, then the data must still be prepared for Data Analysts and end users using view logic.
Summary
In this way, the zero ETL approach actually ensures less effort when integrating the data and, above all, can also result in cost advantages due to less duplicate data storage and, if necessary, no additional tools. In order to make the data usable for use cases in the end, however, efforts are still necessary.
