Spark ETL Chapter 9 with Lakehouse | Apache Iceberg

Previous blog/Context:

In an earlier blog, we discussed Spark ETL with Lakehouse (with HUDI). Please find below blog post for more details.

Introduction:

In this blog, we will discuss Spark ETL with Apache iceberg. We will first understand what Apache iceberg is and why to use Apache iceberg for creating a Lake house. We will source data from one of the source systems which we have learned till now and load that data into Apache iceberg format. We will create an on-premise lake house and load all data into it.

What is Apache Iceberg?

Apache Iceberg is an open-source table format that aims to provide a more scalable and efficient way of managing large, complex datasets in big data environments. It was originally developed at Netflix to address some of the limitations of existing data storage formats, such as Apache Parquet and Apache ORC.

Iceberg is designed to support features that are common in modern data systems, such as schema evolution, ACID transactions, and time travel (the ability to query data as it existed at some point in the past). It accomplishes this by separating the table’s data files (which contain the actual data) from its metadata (which describes the table’s schema, partitions, and other properties).

Iceberg uses several performance optimizations to improve query performance, such as column-level statistics and Bloom filters. It also integrates with a variety of big data processing engines, including Apache Spark, Apache Hive, and Presto.

Overall, Apache Iceberg is a powerful and flexible tool for managing large, complex datasets in big data environments. It provides several advanced features that make it well-suited to a wide range of use cases, including data warehousing, analytics, and machine learning.

Why use Apache Iceberg?

There are several reasons why you might want to use Apache Iceberg:

• Scalability: Iceberg is designed to scale petabytes of data, making it an ideal choice for large data sets.
• Flexibility: Iceberg is designed to be flexible, making it easy to add or remove data without having to rewrite the entire table.
• Performance: Iceberg is designed to provide a balance between performance and flexibility, making it an ideal choice for data warehousing and analytics workloads.
• Open-source: Iceberg is an open-source project, which means that it is free to use and can be customized to meet your specific needs.

Today, we will be doing the operations below ETL and with this, we will also be learning about the Apache iceberg and how to build a lake house.

1. Read data from MySQL server into Spark
2. Create a HIVE temp view from a data frame
3. Load filtered data into iceberg format (create initial table)
4. Load filtered data again into iceberg format in the same table
5. Read iceberg tables using Spark data frame
6. Create Temp HIVE of delta tables
7. Explore data

First, clone below GitHub repo, where we have all the required sample files and solution

If you don’t have a setup for Spark instance follow the earlier blog for setting up Data Engineering tools in your system. (Data Engineering suite will setup Spark, MySQL, PostgreSQL, and MongoDB in your system) In that Spark instance, we already have packages installed for Azure blog storage and Azure Data Lake Services.

Start the Spark application with all required packages

First, we will start the Spark session with all the required packages and configurations for Apache Iceberg. We know that with our spark instance, we don’t have packages (jar file) available for Apache iceberg, so when we start the spark session, we need to externally specify that. We will also be using MySQL so we will specify package requirements for MySQL also.

With Apache Iceberg, we also need to pass the configurations below.

By default, warehouse location is $PWD (it means the location where we start the spark session) but here we are giving warehouse so it will create a folder in our folder and put all tables inside it.

Now, we have our spark session available with all the required packages and configuration, so we can start the ETL process.

Read data from MySQL server into Spark

(If you have already completed chapter 7, you can skip reading data from MySQL and create the HIVE table, and can directly go to create HUDI table section)

For this ETL, we are also using the same MySQL as the source system and are loading the same table. We will not discuss much on how to load data from MySQL and how to create a HIVE table as we have already discussed in detail in Chapter 7.

If you don’t have already uploaded data into MySQL, please follow the earlier blog for the same.

We will read this data from Spark and we will create a spark data frame and HIVE table on this.

Create a HIVE temp view from a data frame

We will create a HIVE temp view from a data frame.

We will explore data and check the highest food group and filter with one group.

Load filtered data into iceberg format (create initial table)

We have data frame “newdf” available in which we have only one group food is there. We will use that to create the first iceberg table.

This will create a folder with the name “iceberg_food” inside the warehouse folder.

Inside “Iceberg_food” folder, has all the data and metadata. It has two separate folders for data and metadata as below.

In the data folder, it creates a parquet file that stores all data.

And in the metadata folder, has all metadata files.

Metadata stored in JSON file as below

Load filtered data again into iceberg format in same table

We will create one more data frame by filtering one more food group and then appending data to the same iceberg table.

Now, if we check folders, we will see more files for data and metadata.

For metadata

Read iceberg tables using Spark data frame

Now, we will read iceberg tables into the Spark data frame.

Create Temp HIVE of delta tables

We will create a HIVE temp view from a data frame. So that we can write Spark SQL and explore data available in Iceberg format.

Explore data

We can Spark SQL queries and explore data.

Conclusion:

Here, we have learned the concepts below.

• Understating of Apache Iceberg
• How to install Iceberg packages from the Maven repo
• How to configure Spark parameters for Iceberg
• How to create an Iceberg table and load data
• How data is stored in Iceberg format
• How to read data from Iceberg tables
• How to write Spark SQL queries on Iceberg

Video Explanation:

Level Up Coding

Thanks for being a part of our community! Before you go:

• 👏 Clap for the story and follow the author 👉
• 📰 View more content in the Level Up Coding publication
• 💰 Free coding interview course ⇒ View Course
• 🔔 Follow us: Twitter | LinkedIn | Newsletter

🚀👉 Join the Level Up talent collective and find an amazing job