BigQuery now supporting Query Queues

Using Query Queues for Concurrency in Google BigQuery

Google just announced that Query queues are now available in preview for on-demand and flat-rate customers. After you enable query Google BigQuery automatically determines the query concurrency rather than setting a fixed limit. Flat rate customers have the possibility to override this setting with a custom concurrency target. Additional queries beyond the concurrency target are queued until processing resources become available [1].

When building up and using a modern SaaS-based Data Warehouses the ability to provide a lot of data to many users and enable concurrent queries is one of the most important factors. The newer often cloud-based Data Warehouse technologies solve the problem of high simultaneity through good scalability and enable companies to perform data analysis on a large scale. Normally BigQuery limits the number of concurrently running interactive queries to 100. Queries that exceed this will return a quota error.

With the new feature query queues are enabled, BigQuery dynamically determines the query concurrency based on available compute resources. Optionally, you can specify the target concurrency for a reservation to ensure that each query is allocated some minimum number of slots. Additional queries wait in a queue until there is enough capacity available to begin execution [2]:

An important criteria for Data Warehouses is that many users are able to work simultaneously. While managing a few users is easy to manage, scaling to thousands is very hard to manage. Everyone must be able to work with the same real-time data without negatively impacting other users and no users like to get an quota error. And this is exactly where Google now starts by enabling query queues for BigQuery. Users may have to wait a little longer, but they won’t get any errors. In my opinion, this is a very useful function extension. Remember that it is, as I said. in the Preiew, but usually the features then also come promptly for all and generally available.

If your are a Google BigQuery user, fan or whatever the following lately updates may be also interesting for you:

• Google improves Data Security in it’s Data Warehouse BigQuery — Using column level SQL encryption with Cloud KMS keys
• Using SQL Translation in BigQuery for easy Data Warehouse Migration — Google’s Data Warehouse now supports most major SQL Dialects
• Improved Storage Read API Quotas in Google BigQuery — How Google empowers it’s Data Warehouse even more

Sources and Furhter Readings

[1] Google, BigQuery Release Notes (2022)

[2] Google, Use query queues (2022)