Google launches Query Queues for BigQuery

How Google wants to improve Concurrency of their Data Warehouse

Already last year, Google has announced Query Queues for its SaaS Data Warehouse service BigQuery. Now, this feature is finally generally available.

Although it is not entirely new because, as mentioned, it is already available in preview, it has to be said that it is a hindrance for a productive environment, since you do not want to use and implement anything that might be retracted.

With query queues, BigQuery automatically determines your query concurrency based on available slots rather than a fixed limit. Once the maximum concurrency is reached, additional queries are queued until processing resources are available[1].

Concurrency is an important criteria for a Data Warehouse, since many users are able to work simultaneously. While managing a few users is easy to manage, scaling to thousands is very hard to manage. Everyone must be able to work with the same real-time data without negatively impacting other users and without receiving any quota error.

Therefore, Google has rolled out query queues and enabled them by default over the last several weeks. Also, no user action is required and Google has promised no degradation in your query performance. Optionally, you have the possibility to set the maximum concurrency target for a reservation. Furthermore, you can also control the interactive and batch query queue timeout by using default configurations[1].

Sources and Further Readings

[1] Google, BigQuery release notes (2023)

[2] Google, Use query queues (2022)