Google launches better Cross Cloud Operations for BigQuery

How Google BigQuery now lets you filter Data with Insert- Into-Select

Google continues to expand its integration of BigQuery with other systems such as AWS S3 and Azure Blob Storage. For some time now, users have been able to use Google BigLake to analyze platform-independent data using a zero ETL approach or to integrate it using a fixed process.

Now, Google has updated the INSERT INTO SELECT statement. You can now filter data from files in Amazon S3 and Azure Blob Storage and append it into BigQuery tables[1].

So if you want to realize your data integration via SQL and via BigLake in BigQuery, which is of course a simple practical application case as it is easy to implement, this command of will offer you some new possibilities. You can store a command logic in BigQuery for the respective table or file in the S3 or Azure storage. The logic you would use for this is down below[2]:

SELECT
  table_name, ddl
FROM
  `myproject`.myawsdataset.INFORMATION_SCHEMA.TABLES
WHERE
  table_name = 'orders';

For further information, example code and limitations please use the official Google source linked below.

It will be interesting to see how this topic develops and whether other cloud providers and services will soon be integrated in this way, but this change should certainly mean easier data integration for Data Engineers.

Sources and Further Readings

[1] Google, BigQuery release notes (2023)

[2] Google, Load data with cross-cloud operations (2023)