avatarChristianlauer

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

1929

Abstract

irectly from source systems in near real time [1].</p><figure id="cba1"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/0*Uv1gCwYisalk6pv2"><figcaption>Architecture of Data Stream — Image Source: <a href="https://cloud.google.com/datastream-for-bigquery">Google</a>[1]</figcaption></figure><p id="d935">This of course brings some advantages like near real-time insights in BigQuery, serverless ELT/ETL data pipelines that scales automatically, with no resources to provision or manage. Google Datastream also ensures source schemas change. Datastream seamlessly handles schema drift and automatically replicates new columns and tables added in the source to BigQuery. For now it’s available for: [1]</p><ul><li>MySQL</li><li>PostgreSQL</li><li>AlloyDB</li><li>Oracle databases</li></ul><p id="e350">The service is still in preview but it seems that everyone also without a company account can already try it out — you just have to enable the API.</p><figure id="2a91"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*DxazC5BZMynrFYHmhBq1eg.png"><figcaption>Enable BigQuery API — Image by Author</figcaption></figure><p id="1528">Afterwards, you can then create your first stream into Google Data Warehouse BigQuery the process is relatively straight forwards, also find the documentation and a video in the Google Source below.</p><figure id="d687"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*chJA9BSCc9glaVNVVzKUOg.png"><figcaption>Creating a Stream — Image by Author</figcaption></figure><p id="d13d">After Google has already brought out some interesting innovations this year in the area of data analytics and especially BigQuery, now follows a feature that many of us have been eagerly awaiting. If the source could not be accessed via Data Transfer Service or the new service BigLake via Cloud Storage, a more complex replication via Dataflow or a third party provider

Options

like talend and co. was needed.</p><div id="a5cb" class="link-block"> <a href="https://readmedium.com/google-launches-new-data-lakehouse-engine-big-lake-4648d6be458e"> <div> <div> <h2>Google launches new Data Lakehouse Engine — Big Lake</h2> <div><h3>How it’s great News for all Google Cloud Platform Users</h3></div> <div><p>medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*s1M8ondCwffP3uExWjLhzw.jpeg)"></div> </div> </div> </a> </div><p id="c9f8">Here, you can simply integrate relational databases via SaaS CDC, so to speak. This should ensure that Data Engineers and Data Scientists spend significantly less time on data integration and can concentrate on more value-adding activities.</p><p id="3585">For other data sources like BigTable, Google has also recently simplified data integration via Zero ETL Approach — read more here:</p><div id="c5f3" class="link-block"> <a href="https://readmedium.com/read-data-directly-with-bigquery-sql-with-the-zero-etl-approach-a2dc34ff76ff"> <div> <div> <h2>Read Data directly with BigQuery SQL with the Zero — ETL Approach</h2> <div><h3>No more ETL needed in between BigQuery and BigTable</h3></div> <div><p>medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*ToYrdSf4RiZYWiQEnXCkDQ.jpeg)"></div> </div> </div> </a> </div><h2 id="9069">Sources and Further Readings</h2><p id="77c6">[1] Google, <a href="https://cloud.google.com/datastream-for-bigquery">Datastream for BigQuery</a> (2022)</p></article></body>

Google launches new Data Service Datastream

New Tool for Seamless Replication from Databases to BigQuery

Photo by Jess Barnett on Unsplash

Google already offers data integration options for BigQuery with Dataflow, Data Prep and the Data Transfer Service, but with Datastream it should now become even easier.

While Data Flow needs a lot of programming and Dataprep is more used for data preparation and the Data Transfer Service only offers some data sources, Datastream is supposed to be a simple replication service for relational databases.

Therefore, Datastream uses BigQuery’s Change Data Capture (CDC) functionality and Storage Write API to efficiently replicate updates directly from source systems in near real time [1].

Architecture of Data Stream — Image Source: Google[1]

This of course brings some advantages like near real-time insights in BigQuery, serverless ELT/ETL data pipelines that scales automatically, with no resources to provision or manage. Google Datastream also ensures source schemas change. Datastream seamlessly handles schema drift and automatically replicates new columns and tables added in the source to BigQuery. For now it’s available for: [1]

  • MySQL
  • PostgreSQL
  • AlloyDB
  • Oracle databases

The service is still in preview but it seems that everyone also without a company account can already try it out — you just have to enable the API.

Enable BigQuery API — Image by Author

Afterwards, you can then create your first stream into Google Data Warehouse BigQuery the process is relatively straight forwards, also find the documentation and a video in the Google Source below.

Creating a Stream — Image by Author

After Google has already brought out some interesting innovations this year in the area of data analytics and especially BigQuery, now follows a feature that many of us have been eagerly awaiting. If the source could not be accessed via Data Transfer Service or the new service BigLake via Cloud Storage, a more complex replication via Dataflow or a third party provider like talend and co. was needed.

Here, you can simply integrate relational databases via SaaS CDC, so to speak. This should ensure that Data Engineers and Data Scientists spend significantly less time on data integration and can concentrate on more value-adding activities.

For other data sources like BigTable, Google has also recently simplified data integration via Zero ETL Approach — read more here:

Sources and Further Readings

[1] Google, Datastream for BigQuery (2022)

Data Science
Google
Technology
Big Data
Bigquery
Recommended from ReadMedium