Snowflake launches Streams on Views
How this can ease your Data CDC Pipelines and Data Integration
With this new function, you can now enable CDC using views, which can greatly simplify data integration and subsequent data analysis.
What are Streams again?
Snowflake operates streams as the following: it creates an initial snapshot of each row within the sources object in a logical manner (e.g. a table, an external table, or the underlying tables for a view). The change tracking system utilized by the stream then records information about the DML changes after this snapshot was taken. Change records provide the state of a row before and after the change. Change information mirrors the column structure of the tracked source object and includes additional metadata columns that describe each change event [1].
How Streams on Views work
Now Snowflake has announced that you can use streams on views with extended table streams to track change data capture (CDC) records for views, including secure views.
This new feature is however limited to views that meet the following requirements [1]:
- All of the underlying tables must be native tables.
- Projections
- Filters
- Inner or cross joins
- UNION ALL
At this time, Materialized views are not supported.
Summary
A feature that will surely simplify ETL or ELT processes within Snowflake and make Data Engineers happy. If you want to get further into the topic, the following article might be also interesting for you:
Sources and Further Readings
[1] Snowflake, Introduction to Streams (2023)
