avatarChristianlauer

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

1992

Abstract

the good connectivity in the Google Cloud, the services can easily exchange data with each other and thus be used for analysis, machine learning and other topics.</p><figure id="ca77"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/0*UaKRjip0OH4X3cxM.jpg"><figcaption>Data Lakehouse on GCP — Source: <a href="https://cloud.google.com/blog/products/data-analytics/open-data-lakehouse-on-google-cloud">Google</a> [2]</figcaption></figure><p id="69ab">So everything that goes in the direction of unstructured and semi-structured files, you can store well with cloud storage. Tables can be stored directly in BigQuery, although it must be said that BigQuery is now also a hybrid between SQL and NoSQL database and allows, for example, the storage as data type JSON.</p><div id="4e9f" class="link-block"> <a href="https://readmedium.com/amazing-new-feature-google-bigquery-now-supports-json-as-a-data-type-39e0464d4c43"> <div> <div> <h2>Google BigQuery now supports JSON as a Data Type</h2> <div><h3>How to use JSON as a Data Type in BigQuery SQL</h3></div> <div><p>medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*O8ZnjcjUy5UL86bI3qgxBg.jpeg)"></div> </div> </div> </a> </div><h2 id="6cc8">Google takes it to the extreme with BigLake</h2><p id="3d8e">Google even goes one step further and offers platform-independent data analysis via BigLake. So you can access storage like S3 and co. via cloud storage and on the basis of BigQuery Omni and perform BigQuery SQL analyses. This has the advantages that you don’t have to transport data and pay for duplicate storage and that even as an AWS or Azure user you can use the very cool data analytic tools from Google.</p><div id="cb02" class="link-block"> <a href="https://readmedium.com

Options

/google-launches-new-data-lakehouse-engine-big-lake-4648d6be458e"> <div> <div> <h2>Google launches new Data Lakehouse Engine — Big Lake</h2> <div><h3>How it’s great News for all Google Cloud Platform Users</h3></div> <div><p>medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*s1M8ondCwffP3uExWjLhzw.jpeg)"></div> </div> </div> </a> </div><h2 id="16af">Summary</h2><p id="2964">Google offers all the tools necessary to build a modern data platform and with Google BigLake even offers a bit more than others. But this architecture in general is of course also possible with other providers such as AWS or MS Azure. Microsoft for example offers with Azure Synapse Analytics such an analysis platform.</p><div id="3045" class="link-block"> <a href="https://christianlauer90.medium.com/building-a-data-lakehouse-with-azure-synapse-a908316a3994"> <div> <div> <h2>Building a Data Lakehouse with Azure Synapse</h2> <div><h3>How to build a modern Data Platform in the Microsoft Cloud</h3></div> <div><p>christianlauer90.medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*SOOdfJda8wSM9FYFVVg2iA.jpeg)"></div> </div> </div> </a> </div><h2 id="e1ee">Sources and Further Readings</h2><p id="4b1d">[1] AWS, <a href="https://aws.amazon.com/big-data/datalakes-and-analytics/data-lake-house/?nc1=h_ls">What is a Lake House approach?</a> (2021)</p><p id="b9d0">[2] Google Cloud, <a href="https://cloud.google.com/blog/products/data-analytics/open-data-lakehouse-on-google-cloud">Open data lakehouse on Google Cloud</a> (2021)</p></article></body>

Build a Data Lakehouse on Google Cloud

How to use Cloud Storage and BigQuery to build up a modern Data Platform

Photo by Jingda Chen on Unsplash

The Data Lakehouse is a modern approach to designing data platforms, it combines Data Lake and Data Warehouse — how you can build one in the Google Cloud you will learn in the following article.

With Google’s cloud services, you have the right tools at your disposal, but before we look at which services you can use, here’s a recap on the topic of Data Lakehouses.

Recap: The Data Lakehouse Concept

A Data Lakehouse is not just about integrating a Data Lake with a Data Warehouse, but rather integrating a Data Lake, a Data Warehouse, and purpose-built storage to enable unified governance and ease of data movement [1]. From my own experience, it has often shown that Data Lakes can be realized much faster. Once all data is available, Data Warehouses can still be built on top of it as a hybrid solution. Read more about it here.

Hybrid Data Lake Concept — Image from Author

How to build up a Data Lakehouse on GCP?

Let’s take a look at which Google Cloud Services you can use to build such a Data Lakehouse in the Google Cloud. Here, Cloud Storage and BigQuery are used as storage. Due to the good connectivity in the Google Cloud, the services can easily exchange data with each other and thus be used for analysis, machine learning and other topics.

Data Lakehouse on GCP — Source: Google [2]

So everything that goes in the direction of unstructured and semi-structured files, you can store well with cloud storage. Tables can be stored directly in BigQuery, although it must be said that BigQuery is now also a hybrid between SQL and NoSQL database and allows, for example, the storage as data type JSON.

Google takes it to the extreme with BigLake

Google even goes one step further and offers platform-independent data analysis via BigLake. So you can access storage like S3 and co. via cloud storage and on the basis of BigQuery Omni and perform BigQuery SQL analyses. This has the advantages that you don’t have to transport data and pay for duplicate storage and that even as an AWS or Azure user you can use the very cool data analytic tools from Google.

Summary

Google offers all the tools necessary to build a modern data platform and with Google BigLake even offers a bit more than others. But this architecture in general is of course also possible with other providers such as AWS or MS Azure. Microsoft for example offers with Azure Synapse Analytics such an analysis platform.

Sources and Further Readings

[1] AWS, What is a Lake House approach? (2021)

[2] Google Cloud, Open data lakehouse on Google Cloud (2021)

Data Science
Technology
Google
Big Data
Bigquery
Recommended from ReadMedium