3 new Features in Big Query that will make Data Engineers happy

Information Schema now in GA, more Fail Safe Information and BigLake Metastore

Yesterday (14.06.23), Google has released three interesting updates in the area of Data Engineering for their Data Warehouse BigQuery.

Firstly, Google BigQuery now provides information about the fail-safe period, at least in preview for now. The fail-safe period offers an additional seven days of data storage after the time travel window, so that the data is available for emergency recovery[1]. Read more about the time travel function in the article below:

Another great new feature that is already generally available now and allows Data Engineers to better analyze capabilities of data usage are INFORMATION_SCHEMA views that show table storage metadata . Here, you can use the TABLE_STORAGE view to get a snapshot of current storage usage for tables at the project level and the TABLE_STORAGE_BY_ORGANIZATION view to get a snapshot of current storage usage for tables at the organization level[1].

Last but not least, if you are working with Apache Iceberg tables — BigLake Metastore is now generally available[1]. You can use this to access and manage Iceberg table metadata from multiple sources. This should also simplify the handling of data from this system and provide better insights. With BigLake, Google has already created a service last year with which platform-independent data analysis is possible and they are continuously expanding it.

Sources and Further Readings

[1] Google, BigQuery release notes (2023)