Data Lakehouse vs. Data Lake
What are the Differences and how they are build up on each other
With the replacement of the classic Data Warehouse by new modern often cloud-based systems such as Data Lakes, certain problems are occuring. Because a Data Lake is a large container of all possible and often still raw data, these can not be used well for e.g. Self Service BI tools. This is where the Data Lakehouse comes into play. Data Lakehouses are a mixture of Data Lakes and classical Data Warehouses.
Data Warehouses and Data Lakes
Data Lakes and Data Warehouses are established terms when it comes to storing Big Data, but the two terms are not synonymous. As said before Data Lake is a large pool of raw data for which no use has yet been determined. A Data Warehouse, on the other hand, is a repository for structured, filtered data that has already been processed for a specific purpose [1].
While Data Warehouses use the classic ETL process in combination with structured data in a relational database, a Data Lake uses paradigms such as ELT and a schema on read as well as often unstructured data [2].
So what is a Data Lakehouse?
It is not just about integrating a Data Lake with a Data Warehouse, but rather integrating a Data Lake, a Data Warehouse, and purpose-built storage to enable unified governance and ease of data movement [3]. From my own experience has often shown that a Data Lakes can be realized much faster. Once all data is available, Data Warehouses can still be built on top of it as a hybrid solution.
This makes rigid and classically planned Data Warehouses a thing of the past. This greatly accelerates the provision of dashboards and analyses and is a good step towards a data-driven culture. An implementation with new SaaS services from the cloud and approaches such as ELT instead of ETL also accelerate the development.
The Time for Hybrid Systems
This system architecture makes clouds such as AWS, Cloud or Azure very attractive, because here you can use object storage such as S3, Cloud Storage and others together with classic databases as a data lake and, based on this, often combine it with already existing interfaces with data warehouse technologies such as Google BigQuery, Azure Synapse or AWS Redshift. And the data lakehouse is ready, so to speak. If necessary, companies can even rely only on the latter, because these new data warehouse technologies offer column-based and NoSQL functionalities together with interfaces and query functions on other database systems, so that a data transfer may no longer be necessary. BigQuery Omni is a good example of this — here you can even query data from other cloud platforms.
BigQuery Omni provides a unified management interface through Google Cloud. BigQuery Omni can use your existing Google Cloud account and BigQuery projects. You can write a standard SQL query in the Cloud Console to query data in AWS or Azure, and see the results displayed in the Cloud Console [4].
Summary
So it’s not so much about Data Lakehouses vs. Data Lakes rather it is that Data Lakehouses are based on Data Lakes. Different SQL and NoSQL databases serve as data storage for the raw data, which can then be processed and analyzed with modern Data Warehouses. Hybrid technologies such as Google BigQuery and other solutions even offer everything from a single source, so that you can access other systems and platforms directly from there using SQL.
Sources and Further Readings
[1] talend, Data Lake vs. Data Warehouse
[2] IBM, Charting the data lake: Using the data models with schema-on-read and schema-on-write (2017)
[3] AWS, What is a Lake House approach? (2021)
[4] Google, What is BigQuery Omni (2022)
