avatarGuillermo Musumeci

Summary

The provided content outlines a comprehensive guide on configuring Azure Databricks Unity Catalog with Terraform, detailing the necessary steps and code examples for setting up SQL warehouses, Unity Catalog components, and integrating them with Azure services.

Abstract

The web content is a detailed technical tutorial that guides readers through the process of setting up Azure Databricks Unity Catalog using Terraform. It covers the prerequisites, such as having an existing Azure Databricks Workspace with a Premium SKU. The guide explains how to configure Terraform service principal permissions, deploy SQL warehouses using both virtual machines and serverless compute options, and configure the Unity Catalog, including creating metastores and assigning permissions. It also demonstrates how to create and manage Azure Storage Accounts and Containers specifically for Unity Catalog, and how to assign the necessary permissions to the Access Connector. The tutorial emphasizes the importance of the Unity Catalog for centralized access control, auditing, lineage, and data discovery across Azure Databricks workspaces. The article concludes by providing the complete code for the configuration on GitHub and encourages reader support for the story.

Opinions

  • The author emphasizes the benefits of using Azure Databricks for processing, storing, cleaning, sharing, analyzing, and monetizing datasets, highlighting its versatility for various data tasks.
  • The tutorial suggests that serverless compute for SQL warehouses is a superior option due to its elasticity and fully managed nature, which provides instant access to compute resources.
  • The author points out the necessity of having a dedicated Azure Storage Account for each Unity Catalog metastore to ensure proper governance and management of data assets.
  • There is an opinion that managing and deploying cloud infrastructure through Terraform provides a streamlined and efficient approach to setting up Azure Databricks Unity Catalog.
  • The article conveys the importance of proper security and governance, advocating for the use of managed identities and role-based access control to protect data assets.
  • The author's note on the deprecation warning when creating a Serverless SQL Warehouse suggests that while the feature is supported, there may be ongoing updates or changes to the API or Databricks Provider that users should be aware of.

How to Configure Azure Databricks Unity Catalog with Terraform Part 1

In this story, we will learn how to configure Azure Databricks Unity Catalog with Terraform.

In particular, we will learn:

  • Deploying SQL Warehouse
  • Creating the Databricks Access Connector for the Unity Catalog
  • Creating the Azure Storage Account for the Unity Catalog
  • Creating the Azure Storage Container for the Unity Catalog
  • Assigning Permissions to the Access Connector to the Unity Catalog Storage Account
  • Creating the Databricks Metastore
  • Creating the Databricks Metastore with a Group Owner
  • Assigning the Metastore to the Workspace
  • Assigning the Access Connector to the Metastore

List of my Azure Databricks-related stories:

1. Intro: What is Azure Databricks, and What is it Used For?

Azure Databricks is a unified, open analytics platform for building, deploying, sharing, and maintaining enterprise-grade data, analytics, and AI solutions at scale.

The Azure Databricks Lakehouse Platform integrates with cloud storage and security in our cloud account and manages and deploys cloud infrastructure on your behalf.

Companies use Azure Databricks to process, store, clean, share, analyze, model, and monetize their datasets with solutions from BI to machine learning.

They use the Azure Databricks platform to build and deploy data engineering workflows, machine learning models, analytics dashboards, and more.

The Azure Databricks workspace provides a unified interface and tools for most data tasks, including:

  • Data processing workflows scheduling, and management
  • Working in SQL
  • Generating dashboards and visualizations
  • Data ingestion
  • Managing security, governance, and HA/DR
  • Data discovery, annotation, and exploration
  • Compute management
  • Machine learning (ML) modeling and tracking
  • ML model serving
  • Source control with Git

2. Intro: Azure Databricks Architecture

Azure Databricks operates out of a control plane and a data plane.

The control plane hosts backend services managed by Azure Databricks, and the data plane is all infrastructure managed by us.

Source: https://learn.microsoft.com/en-us/azure/databricks/getting-started/overview
  • The control plane includes the backend services Azure Databricks manages in its Azure account. Notebook commands and other workspace configurations are stored in the control plane and encrypted at rest.
  • Our Azure account manages the data plane and is where our data resides. This is also where data is processed. Use Azure Databricks connectors to connect clusters to external data sources outside of our Azure account to ingest data, or for storage. We can also ingest data from external streaming data sources, such as events data, streaming data, IoT data, and more.
  • Our data is stored at rest in our Azure account in the data plane and in our own data sources, not the control plane, so we maintain control and ownership of your data.

3. Prerequisites

An existing Azure Databricks Workspace is required.

Check the story “How to Deploy Databricks in Azure with Terraform Step by Step” to deploy one Databricks Workspace.

The Workspace SKU needs to be Premium.

4. Configuring Terraform Service Principal Permissions

Log to https://accounts.azuredatabricks.net/users/serviceprincipals

And we need to check if our Terraform Service Principal is a member of the Account admin role.

If not, we need to click on the Service Principal's name, then on the Roles tab, and enable the Account admin role.

5. Configuring the Terraform Providers

To execute this code, we must define two providers: Azure and Databricks Providers.

First, we need to define provider variables in the “variables.tf” file:

# ------------------------
# Authentication Variables
# ------------------------

variable "aad_tenant_id" {
  type        = string
  description = "The id of the Azure Tenant to which all subscriptions belong"
}

variable "aad_subscription_id" {
  type        = string
  description = "The id of the Azure Subscription"
}

variable "aad_client_id" {
  type        = string
  description = "The client id of the Service Principal for interacting with Azure resources"
}

variable "aad_client_secret" {
  type        = string
  description = "The client secret of the Service Principal for interacting with Azure resources"
  sensitive   = true
}

# --------------------
# Databricks Variables
# --------------------

variable "databricks_workspace_rg" {
  type        = string
  description = "The existing Resource Group where Databricks Workspace was deployed"
}

variable "databricks_workspace_id" {
  type        = string
  description = "The ID of the Databricks Workspace for this deployment"
}

variable "databricks_workspace_url" {
  type        = string
  description = "The URL of the Databricks Workspace for this deployment"
}

And, then we will use the “provider.tf” file to define all providers:

# Define Terraform provider
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
    }
    databricks = {
      source = "databricks/databricks"
    }
  }
}

# Define the Azure provider
provider "azurerm" {
  client_id       = var.aad_client_id
  client_secret   = var.aad_client_secret
  tenant_id       = var.aad_tenant_id
  subscription_id = var.aad_subscription_id
  features {}
}

# Define the Databricks Workspace provider
provider "databricks" {
  host                = var.databricks_workspace_url
  azure_client_id     = var.aad_client_id
  azure_tenant_id     = var.aad_tenant_id
  azure_client_secret = var.aad_client_secret
}

Finally, we will use the “terraform.tfvars” file to configure the providers:

databricks_workspace_rg  = "kopicloud-dbx-dev-we-rg"
databricks_workspace_id  = "1420239596697491"
databricks_workspace_url = "https://adb-142023959669749.1.azuredatabricks.net/"

Note: Use the output of the Databricks Workspace Terraform code execution to get these values.

6. Deploying SQL Warehouse

An SQL warehouse is a compute resource that lets us run SQL commands on data objects within Databricks SQL. Compute resources are infrastructure resources that provide processing capabilities in the cloud.

We will use the resource “databricks_sql_global_config” to configure the security policy, databricks_instance_profile, and data access properties for all databricks_sql_endpoint of the Databricks Workspace and the resource “databricks_sql_endpoint” to create the SQL Warehouse.

We can deploy an SQL Warehouse that runs on virtual machines (default) or a Serverless SQL Warehouse.

6.1. Creating a SQL Warehouse

Deploy a SQL Warehouse using virtual machines.

Note: depending on our version of the Azure Provider, we may need to add the line enable_serverless_compute = true to the resource “databricks_sql_global_config”

resource "databricks_sql_global_config" "this" {
  data_access_config = {}
}

resource "databricks_sql_endpoint" "this" {
  name = "kopicloud-sql-warehouse"
  cluster_size = "2X-Small"
  min_num_clusters = 1
  max_num_clusters = 1
  auto_stop_mins = 10
  warehouse_type = "PRO"

  depends_on = [databricks_sql_global_config.this]
}

6.2. Creating a Serverless SQL Warehouse

With serverless compute, the compute layer exists in our Azure Databricks account rather than our Azure subscription. This gives users in our account instant access to fully managed and elastic compute resources. Serverless compute is supported for use with Databricks SQL. Serverless compute does not affect how Databricks Runtime clusters work with notebooks and jobs.

Note: Serverless SQL Warehouses do not have public IP addresses.

Requirements:

resource "databricks_sql_global_config" "this" {
  data_access_config = {}
  enable_serverless_compute = true
}


resource "databricks_sql_endpoint" "this" {
  name = "Standard"
  cluster_size = "2X-Small"
  min_num_clusters = 1
  max_num_clusters = 1
  auto_stop_mins = 10
  warehouse_type = "PRO"
  
  enable_serverless_compute = true
  depends_on = [databricks_sql_global_config.this]
}

Note: when you run the code to create a Serverless SQL Warehouse, we will receive a “Warning: Argument is deprecated” message. The resource is created. Seems like an issue on the API or Databricks Provider.

6.3. Validating SQL Warehouse Status

We open the Microsoft Azure Databricks console using our Databricks Workspace URL. Click on the SQL Warehouses menu on the left.

7. Configuring the Unity Catalog

The Unity Catalog provides centralized access control, auditing, lineage, and data discovery capabilities across Azure Databricks workspaces.

Source: https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/

7.1. The Unity Catalog Object Model

In Unity Catalog, the hierarchy of primary data objects flows from the metastore to table or volume:

  • Metastore: The top-level container for metadata. Each metastore exposes a three-level namespace (catalog.schema.table) that organizes our data.
  • Catalog: The first layer of the object hierarchy is used to organize our data assets.
  • Schema: Also known as databases, schemas are the second layer of the object hierarchy and contain tables and views.
  • Volume: Volumes sit alongside tables and views at the lowest level of the object hierarchy and provide governance for non-tabular data.
  • Table: Tables and views are at the lowest level in the object hierarchy.

7.2. Creating the Databricks Access Connector for the Unity Catalog

The Databricks Access Connector provides Unity Catalog permissions to access and manage data in the storage account.

resource "azurerm_databricks_access_connector" "unity" {
  name                = "kopicloud-dbx-unity-mi"
  location            = var.location
  resource_group_name = var.databricks_workspace_rg

  identity {
    type = "SystemAssigned"
  }
}

7.3. Creating the Azure Storage Account for the Unity Catalog

An Azure storage account is the default storage location for managed tables in the Unity Catalog.

Note: we need to use a dedicated account for each metastore.

resource "azurerm_storage_account" "unity_catalog" {
  name                     = "kopiclouddbxunityst"
  location                 = var.location
  resource_group_name      = var.databricks_workspace_rg
  account_tier             = "Standard"
  account_replication_type = "GRS"
  is_hns_enabled           = true
}

7.4. Creating the Azure Storage Container for the Unity Catalog

A Unity Catalog Container inside the Azure Storage Account is required.

resource "azurerm_storage_container" "unity_catalog" {
  name                  = "unity-catalog-container"
  storage_account_name  = azurerm_storage_account.unity_catalog.name
  container_access_type = "private"
}

7.5 Assigning Permissions to the Access Connector to the Unity Catalog Storage Account

In this final step, we will assign the “Storage Blob Data Contributor” role to the Access Connector.

resource "azurerm_role_assignment" "unity_catalog" {
  scope                = azurerm_storage_account.unity_catalog.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_databricks_access_connector.unity.identity[0].principal_id
}

8. Creating the Databricks Metastore

A metastore is the top-level container of objects in the Unity Catalog.

It stores metadata about data assets (tables and views) and the permissions governing access.

Azure Databricks account admins should create one metastore for each region they operate and assign them to Azure Databricks workspaces in the same region. For a workspace to use Unity Catalog, it must have a Unity Catalog metastore attached.

Each metastore is configured with a managed storage location in our Azure account's Azure Data Lake Storage Gen2 container.

8.1. Creating the Databricks Metastore

Unity Catalog offers a new metastore with built-in security and auditing. This is distinct from the metastore used in previous versions of Databricks (based on the Hive Metastore).

In the code below, the owner will be assigned to the Terraform service principal. In Unity Catalog all users initially have no access to data. Only the Metastore Owner/Admins can create objects and can grant/revoke access on individual objects to users and groups. We will need to use the “databricks_grants” resource to assign permissions.

resource "databricks_metastore" "primary" {
  provider = databricks.workspace
  
  name         = "primary"
  storage_root = format("abfss://%s@%s.dfs.core.windows.net/", 
      azurerm_storage_container.unity_catalog.name, 
      azurerm_storage_account.unity_catalog.name)
  force_destroy = true
  region        = var.location

  depends_on = [ 
    azurerm_storage_container.unity_catalog, 
    azurerm_storage_account.unity_catalog 
  ]
}

8.2. Creating the Databricks Metastore with a Group Owner

The code below is similar to step 8.1. However, we will create a Databricks Unity Catalog Group and assign it as a metastore owner, so we don’t need to grant permissions to each service or resource. The user needs to be a member of the Owner Group.

Note: check the How to Configure Azure Databricks Unity Catalog with Terraform Part 2 story to learn about Databricks groups.

The process is a little bit more complex, so let's take a look in detail:

First, we will create a Databricks group:

// Create a Databricks UC Group for Unity Catalog Admins
resource "databricks_group" "uc_admins" {
  provider = databricks.account

  display_name = "Unity Catalog Admins"
  
  workspace_access      = true
  databricks_sql_access = true
  
  allow_cluster_create       = true
  allow_instance_pool_create = true
}

Then, we will add the Terraform Service Principal to the Databricks group:

// Reference to the Terraform Service Principal
data "databricks_service_principal" "terraform_spn" {
  provider = databricks.account

  application_id = var.aad_client_id
}

// Add Terraform User to the uc_admins group
resource "databricks_group_member" "terraform" {
  provider = databricks.account

  group_id  = databricks_group.uc_admins.id
  member_id = data.databricks_service_principal.terraform_spn.id
}

After, we will create the metastore using the owner parameter:

// Create the Databricks Metastore
resource "databricks_metastore" "primary" {
  provider = databricks.workspace

  name         = "primary"
  storage_root = format("abfss://%s@%s.dfs.core.windows.net/", azurerm_storage_container.unity_catalog.name, module.storage_account_unity_catalog.name)
  force_destroy = true
  region        = var.location
  owner         = databricks_group.uc_admins.display_name

  depends_on = [ 
    module.storage_account_unity_catalog,
    azurerm_storage_container.unity_catalog
  ]
}

This is the result:

8.3. Assigning the Metastore to the Workspace

A single databricks_metastore can be shared across Databricks workspaces, and each linked workspace has a consistent view of the data and a single set of access policies.

We can only create a single metastore for each region our organization operates.

resource "databricks_metastore_assignment" "primary" {
  workspace_id         = var.dbx_workspace_id
  metastore_id         = databricks_metastore.primary.id
  default_catalog_name = "hive_metastore"

  depends_on = [ databricks_metastore.primary ]
}

8.4. Assigning the Access Connector to the Metastore

Each Databricks Metastore requires a Managed Identity that Unity Catalog will assume to access data.

resource "databricks_metastore_data_access" "primary" {
  metastore_id = databricks_metastore.primary.id
  name         = "primary"
  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.unity.id
  }

  is_default = true

  depends_on = [ 
    azurerm_databricks_access_connector.unity, 
    databricks_metastore.primary, 
    databricks_metastore_assignment.primary
  ]
}

8.5. Validation of Databricks Metastore Deployment

We open the Azure Databricks Workspace URL, click on our email address (top, right), and then click in the Manage Account menu.

We click on the Data menu to list all the Metastores.

How to Configure Azure Databricks Unity Catalog with Terraform Part 2 is available here and How to Configure Azure Databricks Unity Catalog with Terraform Part 3 is available here.

The complete code to configure Azure Databricks using Terraform is available at https://github.com/KopiCloud/terraform-azure-databricks-unity-catalog

And that’s all, folks. If you liked this story, please show your support by 👏 this story. Thank you for reading!

Databricks
Unity Catalog
Terraform
Azure
Azure Databricks
Recommended from ReadMedium
avatarRakesh Dey
Row Level Security Implementation in Databricks Table

Databricks has recently made GA Row Filters and Column Masks in Unity Catalog. With Unity Catalog, we can use standard SQL functions to…

3 min read
avatarMark Tinderholt
Setting Up Azure Observability with Log Analytics and Azure Storage

4 min read
avatarPrem Vishnoi(cloudvala)
Databricks Cost Optimization using 6 Method

Choose Optimal Resources:

3 min read
avatarKaushal Akoliya
How to Create a Unity Catalog in Azure Databricks

This article explains how to configure and use Unity Catalog to manage data in your Azure Databricks workspace. It is intended primarily…

6 min read
avatarThe MLOps Guy
Databricks Lakehouse Monitoring: A Practical Hands-On Guide

Databricks Lakehouse Monitoring lets you monitor the statistical properties and quality of the data in all of the tables in your databricks…

9 min read
avatarMatt Weingarten
Databricks Q4 Roadmap: W2W4

3 min read