Free AI web copilot to create summaries, insights and extended knowledge, download it at here

Abstract

you may have used solutions like Cloud Functions, Data Flow or Data Prep as another intermediary.<figure id="11df"><img src="https://cdn-images-1.readmedium.com/v2/resize:fit:800/1*3QLRI0Brn7pA2UChNIBxFg.png"><figcaption>Architecture Example within GCP — Image Source: <a href="https://cloud.google.com/architecture/streaming-avro-records-into-bigquery-using-dataflow?hl=de">Google</a>[1]</figcaption></figure>Here, you would then probably proceed with an architectural solution like the one above. But now you can also integrate data directly into BigQuery via Pub/Sub[2].A BigQuery subscription within PubSub means that messages will be written directly to an existing BigQuery table as they are received. The solutions has of course some advantages, thus one needs a service less, which means less expenditure, maintenance, possible error and if necessary costs. That should please <a href="https://readmedium.com/what-is-a-data-architect-b7e951cccf3a">Data Architects</a>, <a href="https://readmedium.com/what-skills-does-a-data-engineer-need-f3ab2bb3620d">Data Engineers</a> and ultimately <a href="htt

Options

ps://readmedium.com/the-life-of-a-cio-6e926662d840">CIOs</a> all together. Monitoring and troubleshooting are also much easier.For more and deeper information like quotas and prices you can visit the <a href="https://cloud.google.com/pubsub/docs/bigquery">official documentary of Google</a>.If you using GCP and BigQuery more often like me, you may also be interested in these articles:<ul><li><a href="https://readmedium.com/378a65fdc9c1">BigQuery now supporting Query Queues</a></li><li><a href="https://readmedium.com/8ef3df2af71d">3 Big Updates in Google Data Studio</a></li><li><a href="https://readmedium.com/817191ddc636">Google improves Data Security in it’s Data Warehouse BigQuery</a></li></ul><h2 id="36ad">Sources and Further Readings</h2>[1] Google, <a href="https://cloud.google.com/architecture/streaming-avro-records-into-bigquery-using-dataflow?hl=de">Avro-Datensätze mithilfe von Dataflow in BigQuery streamen</a> (2022)[2] Google, <a href="https://cloud.google.com/bigquery/docs/release-notes#July_28_2022">BigQuery release notes</a> (2022)</article></body>

Google improves BigQuery Data Streaming Capabilities

Better Integration of Google BigQuery and PubSub

If you work with the Google Cloud, it may well be that you realize your data integration with the help of Pub/Sub — especially in the area of streaming. Here is some interesting news now.

Pub/Sub is used for streaming analytics and data integration pipelines to ingest and distribute data. This method is as effective as messaging-oriented middleware for service integration or as a queue to parallelize tasks. If you are using data from mainly streaming data from one source to the SaaS Data Warehouse BigQuery, in this case Google’s Pub/Sub, you may have used solutions like Cloud Functions, Data Flow or Data Prep as another intermediary.

Architecture Example within GCP — Image Source: Google[1]

Here, you would then probably proceed with an architectural solution like the one above. But now you can also integrate data directly into BigQuery via Pub/Sub[2].

A BigQuery subscription within PubSub means that messages will be written directly to an existing BigQuery table as they are received. The solutions has of course some advantages, thus one needs a service less, which means less expenditure, maintenance, possible error and if necessary costs. That should please Data Architects, Data Engineers and ultimately CIOs all together. Monitoring and troubleshooting are also much easier.

For more and deeper information like quotas and prices you can visit the official documentary of Google.

If you using GCP and BigQuery more often like me, you may also be interested in these articles:

Sources and Further Readings

[1] Google, Avro-Datensätze mithilfe von Dataflow in BigQuery streamen (2022)

[2] Google, BigQuery release notes (2022)