Improved Storage Read API Quotas in Google BigQuery

Free AI web copilot to create summaries, insights and extended knowledge, download it at here

1178

Abstract

API provides structured row responses in a paginated fashion appropriate for small result sets.</li><li>Alternatively, using the bulk data export with BigQuery extract jobs that let you export table data to Cloud Storage in a variety of file formats such as CSV or JSON.</li></ul>With the BigQuery Storage Read API you have an alternative option along the advantage that when you use the Storage Read API, structured data is sent over the wire in a binary serialization format. This allows for additional parallelism among multiple consumers for a set of results [1]. Keep in mind that the Storage Read API don’t let you use managing BigQuery resources such as datasets, jobs, or tables.Also, since yesterday (06.06.2022) Storage Read API quotas and limits have been increased to [2]:<ul><li>2,000 concurrent ReadRows calls per project in the US and EU multi-regions and 400 concurrent ReadRows calls in other regions.</li><li>And 5,000 to 25,000 plane requests per user per project per minute</li></ul>Besides this update, another new feature was presented: You can now attach Resource Manager tags to data

Options

sets [2]. With tags you can apply Identity and Access Management policies to resources.So a good start to the week for BigQuery users. These new features and improvements make BigQuery even better at handling data security and availability. Other recent enhancements that may be of interest to you:<ul><li><a href="https://readmedium.com/195a90cc5b85">Google improves Data Security in BigQuery:Using Column based Data Masking in BigQuery and Data Catalog</a></li><li><a href="https://readmedium.com/54c05dbfbc1d">Is Google BigLake Superior of all Data Warehouse and Lake Solution? How Google attacks other Providers like Amazon, Microsoft and Co.</a></li><li><a href="https://readmedium.com/e63d34ee4799">Using Collation in Google BigQuery: How to Compare and Sort Strings easily with SQL</a></li></ul><h2 id="6b1e">Sources and Further Readings</h2>[1] Google, <a href="https://cloud.google.com/bigquery/docs/reference/storage">Use the BigQuery Storage Read API to read table data</a> (2022)[2] Google, <a href="https://cloud.google.com/bigquery/docs/release-notes#June_06_2022">Release Notes</a> (2022)</article></body>

How Google empowers it’s Data Warehouse even more

Good news regarding the BigQuery Read API, Google has now significantly improved it, so you can query more data here. This is of course great if you want to analyze data or e.g. also transfer to other systems, especially in times of Big Data is the processing of vast data amounts in an efficient and parallel way very important.

Normally, you have two options in BigQuery for accessing BigQuery data [1]:

Using record based paginated access by using the tabledata.list or jobs.getQueryResults REST API methods. The BigQuery API provides structured row responses in a paginated fashion appropriate for small result sets.

Alternatively, using the bulk data export with BigQuery extract jobs that let you export table data to Cloud Storage in a variety of file formats such as CSV or JSON.

With the BigQuery Storage Read API you have an alternative option along the advantage that when you use the Storage Read API, structured data is sent over the wire in a binary serialization format. This allows for additional parallelism among multiple consumers for a set of results [1]. Keep in mind that the Storage Read API don’t let you use managing BigQuery resources such as datasets, jobs, or tables.

Also, since yesterday (06.06.2022) Storage Read API quotas and limits have been increased to [2]:

2,000 concurrent ReadRows calls per project in the US and EU multi-regions and 400 concurrent ReadRows calls in other regions.

And 5,000 to 25,000 plane requests per user per project per minute

Besides this update, another new feature was presented: You can now attach Resource Manager tags to datasets [2]. With tags you can apply Identity and Access Management policies to resources.

So a good start to the week for BigQuery users. These new features and improvements make BigQuery even better at handling data security and availability. Other recent enhancements that may be of interest to you: