Free AI web copilot to create summaries, insights and extended knowledge, download it at here

Abstract

lumns and tables added in the source to BigQuery. For now it’s available for: [1]</p><ul><li>MySQL</li><li>PostgreSQL</li><li>AlloyDB</li><li>Oracle databases</li></ul><div id="ef1a" class="link-block"> <a href="https://readmedium.com/google-launches-new-data-service-datastream-88fdef635c48"> <div> <div> <h2>Google launches new Data Service Datastream</h2> <div><h3>New Tool for Seamless Replication from Databases to BigQuery</h3></div> <div><p>medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*JGsSAe3Fk25qipg13R8XVg.jpeg)"></div> </div> </div> </a> </div><h2 id="b037">Update 2: Easier Moving Data from S3 compatible Storage to Cloud Storage</h2><p id="7676">Another handy feature when you need to move data from AWS to Google Cloud. It allows you to use Cloud Storage suitable for running applications written for the S3 API. With this new feature, customers can seamlessly copy data from self-managed object storage to Google Cloud Storage [2]. The new BigLake feature mentioned above may also be of interest here, as it allows you to run data analysis platform-independently and via SQL.</p><h2 id="f9b3">Update 3: Better Dataflow Benchmarks and Pub/Sub Monitoring</h2><p id="c72f">Google announced that they expanded support for PerfKit Benchmarker for benchmarking Dataflow pipelines. You can now test your Dataflow pipelines for performance optimization, capacity planning, r

Options

egression testing and TCO estimation. While also for Pub/Sub monitoring dashboards are now part of the Pub/Sub UI in Google Cloud Console. Pub/Sub users can easily monitor the health of their real-time streaming applications by reading charts of insightful metrics [2].</p><h2 id="8921">Summary</h2><p id="9dd1">So quite a lot of news for Data Engineers working in Google Cloud, which should make your work easier. I was especially happy about the possibility to implement ELT and CDC with Datastream to BigQuery more easily. This has always been a challenge and required other tools, now you can build on simple services integrated with BigQuery, which can save you time and money.</p><div id="72df" class="link-block"> <a href="https://readmedium.com/salary-of-a-bi-data-engineer-c0c6cb77713c"> <div> <div> <h2>Salary of a BI Data Engineer</h2> <div><h3>What does a BI Data Engineer do and what do they earn?</h3></div> <div><p>medium.com</p></div> </div> <div> <div style="background-image: url(https://miro.readmedium.com/v2/resize:fit:320/1*Edh5c81tC0XNWdHYF9VwwA.jpeg)"></div> </div> </div> </a> </div><h2 id="8709">Sources and Further Readings</h2><p id="9a28">[1] Google, <a href="https://cloud.google.com/datastream-for-bigquery">Datastream for BigQuery</a> (2022)</p><p id="0678">[2] Google, <a href="https://cloud.google.com/blog/topics/inside-google-cloud/whats-new-google-cloud">What’s new with Google Cloud </a>(2022)</p></article></body>

3 Great News for Google Cloud Data Engineers

How Google improves Data Integration lately

Google has been giving Data Engineers some joy in recent weeks, with many interesting updates.

Three important updates in particular should provide joy, especially the first update is particularly interesting.

Update 1: New Google Service Datastream

Datastream uses BigQuery’s Change Data Capture (CDC) functionality and Storage Write API to efficiently replicate updates directly from source systems in near real time [1].

Architecture of Data Stream — Image Source: Google[1]

This of course brings some advantages like near real-time insights in BigQuery, serverless ELT/ETL data pipelines that scales automatically, with no resources to provision or manage. Google Datastream also ensures source schemas change. Datastream seamlessly handles schema drift and automatically replicates new columns and tables added in the source to BigQuery. For now it’s available for: [1]

MySQL
PostgreSQL
AlloyDB
Oracle databases

Google launches new Data Service Datastream

New Tool for Seamless Replication from Databases to BigQuery

medium.com

Update 2: Easier Moving Data from S3 compatible Storage to Cloud Storage

Another handy feature when you need to move data from AWS to Google Cloud. It allows you to use Cloud Storage suitable for running applications written for the S3 API. With this new feature, customers can seamlessly copy data from self-managed object storage to Google Cloud Storage [2]. The new BigLake feature mentioned above may also be of interest here, as it allows you to run data analysis platform-independently and via SQL.

Update 3: Better Dataflow Benchmarks and Pub/Sub Monitoring

Google announced that they expanded support for PerfKit Benchmarker for benchmarking Dataflow pipelines. You can now test your Dataflow pipelines for performance optimization, capacity planning, regression testing and TCO estimation. While also for Pub/Sub monitoring dashboards are now part of the Pub/Sub UI in Google Cloud Console. Pub/Sub users can easily monitor the health of their real-time streaming applications by reading charts of insightful metrics [2].

Summary

So quite a lot of news for Data Engineers working in Google Cloud, which should make your work easier. I was especially happy about the possibility to implement ELT and CDC with Datastream to BigQuery more easily. This has always been a challenge and required other tools, now you can build on simple services integrated with BigQuery, which can save you time and money.

Salary of a BI Data Engineer

What does a BI Data Engineer do and what do they earn?

medium.com

Sources and Further Readings

[1] Google, Datastream for BigQuery (2022)

[2] Google, What’s new with Google Cloud (2022)