
How to Deploy Azure OpenAI with Private Endpoint and ChatGPT using Terraform
In this story, we will learn how to deploy Azure OpenAI and ChaptGPT using Terraform in Azure.
The Azure OpenAI Service gives customers access to managed AI services using OpenAI GPT-4, GPT-3, Codex, DALL-E, and Whisper models with Azure's security and enterprise promise.
Azure OpenAI co-develops the APIs with OpenAI, ensuring compatibility and a smooth transition from one to the other.
With Azure OpenAI, customers get the security capabilities of Microsoft Azure while running the same models as OpenAI.
Azure OpenAI offers private networking, regional availability, and responsible AI content filtering.
The code uses Azure Cognitive Services, which will also help deploy other models, such as ada, cabbage, curie, text-embedding-ada, cushman, davinci, etc.
1. Prerequisites
Before running our Terraform code, there are some extra steps required.
Note: steps listed below were required at the moment of writing this article. Maybe are not required when you deploy the code. Check Microsoft documentation or see the error section at the end to identify error caused by these prerequisites.
1.1. Enabling OpenAI in our Azure Subscription
Before running our Terraform code, we must request access to Azure OpenAI in our desired Azure subscription.
Note: Currently, access to this service is granted only by application. We can apply for access to Azure OpenAI by completing the form at https://aka.ms/oai/access.
1.2. Enabling GPT-4
GPT-4 models are the latest available models. Due to high demand, this model series is only available by request. To request access, existing Azure OpenAI customers can apply by filling out this form.
Note: this step was required at the moment of writing this article.
1.3. Enabling Customer Managed Keys
To request access, fill and submit the Cognitive Services & Applied AI Customer Managed Keys and Bring Your Own Storage access request form at https://aka.ms/cogsvc-cmk
Note: this step was required at the moment of writing this article.
2. OpenAI Models
Currently, we have three models for ChaptGT in development: the “gpt-35-turbo”, “gpt-4”, and “gpt-4–32k” models. When deploying these models, we must specify a model version.
For GPT-35-Turbo, depending on the region, only version “0301" is available, for GPT-4 models is the version “0314” and for GPT-4–32k, the version is “0613”.
Note: We can find information about OpenAI models on the Microsoft OpenAI models page.
3. Defining the Azure Provider
First, we will define Azure authentication variables.
We will use a Service Principal with a Client Secret. Check the link below for more info about Azure authentication for Terraform: https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs/guides/service_principal_client_secret
variable "azure_subscription_id" {
type = string
description = "Azure Subscription ID"
}
variable "azure_client_id" {
type = string
description = "Azure Client ID"
}
variable "azure_client_secret" {
type = string
description = "Azure Client Secret"
}
variable "azure_tenant_id" {
type = string
description = "Azure Tenant ID"
}
Then, we will configure Terraform and the Azure provider:
# Define Terraform provider
terraform {
required_version = "~> 1.5"
}
# Configure the Azure provider
provider "azurerm" {
features {}
environment = "public"
subscription_id = var.azure-subscription-id
client_id = var.azure-client-id
client_secret = var.azure-client-secret
tenant_id = var.azure-tenant-id
}
4. Creating a Resource Group for OpenAI Resources
In this step, we will create an Azure Resource Group to store all the Azure resources created by our Terraform code.
We will start defining the location variable in the “variables.tf” file:
# Azure Region
variable "location" {
type = string
description = "The region in which this module should be deployed"
default = "north europe"
}
And then the code to create the resource group code in the “main.tf” file:
# Create Resource Group
resource "azurerm_resource_group" "this" {
name = "kopicloud-openai-rg"
location = var.location
}
5. User Assigned Identity
We are going to add the following code to our “main.tf” file to access information about an existing User Assigned Identity:
# Access information about an existing User Assigned Identity
resource "azurerm_user_assigned_identity" "this" {
name = "${lower(replace(var.company," ","-"))}-${var.app_name}-${var.environment}-${var.shortlocation}-identity"
resource_group_name = azurerm_resource_group.this.name
location = azurerm_resource_group.this.location
tags = {
application = var.app_name
environment = var.environment
}
}
6. Creating a VNET and Subnet for OpenAI
Now, we will create the variables to configure the network:
# VNET CIDR
variable "vnet_address_space" {
type = string
description = "VNET for OpenAI VNET"
}
# Subnet CIDR
variable "subnet_address_space" {
type = string
description = "Subnet for OpenAI Public Endpoint"
}
And then the code to create the VNET and the Subnet:
Note #1: the line private_endpoint_network_policies_enabled = true in the resource “azurerm_subnet”; this setting is used to Enable or Disable network policies for the private endpoint on the subnet.
Note #2: the line service_endpoints = [“Microsoft.CognitiveServices”] is very important. Do not forget it!
# Create the VNET for Private Endpoint
resource "azurerm_virtual_network" "this" {
name = "kopicloud-openai-vnet"
address_space = [var.vnet_address_space]
location = azurerm_resource_group.this.location
resource_group_name = azurerm_resource_group.this.name
}
# Create the Subnet for Private Endpoint
resource "azurerm_subnet" "this" {
name = "kopicloud-openai-subnet"
resource_group_name = azurerm_resource_group.this.name
virtual_network_name = azurerm_virtual_network.this.name
address_prefixes = [var.subnet_address_space]
private_endpoint_network_policies_enabled = true
service_endpoints = ["Microsoft.CognitiveServices"]
}
7. Private DNS Zone for OpenAI
We must register the private zone if we don’t have one for OpenAI.
# Create the Resource Group for DNS Zone
resource "azurerm_resource_group" "openai_dns_zone" {
name = "kopicloud-dns-rg"
location = var.location
}
# Create Private DNS Zone for OpenAI
resource "azurerm_private_dns_zone" "openai" {
name = "privatelink.openai.azure.com"
resource_group_name = azurerm_resource_group.dns_zone.name
}
Or, if we have an existing private DNS zone, we need to use a data resource to access the DNS zone.
# Access the private DNS Zone
data "azurerm_private_dns_zone" "openai_dns_zone" {
name = "privatelink.openai.azure.com"
resource_group_name = "kopicloud-core-dns-rg"
}
8. Creating a Private Endpoint for OpenAI
In this step, we will create the Private DNS Zone Virtual Network Link and the Private Endpoint for OpenAI.
Note: remember to update the private_dns_zone_ids in the code below if you use data to access an existing private DNS zone (current example) or if you create a new private DNS zone.
# Create Private DNS Zone Virtual Network Link
resource "azurerm_private_dns_zone_virtual_network_link" "this" {
name = "kopicloud-openai-vnet-link"
resource_group_name = azurerm_private_dns_zone.openai_dns_zone.resource_group_name
private_dns_zone_name = azurerm_private_dns_zone.openai_dns_zone.name
virtual_network_id = azurerm_virtual_network.this.id
}
# Create the Private Endpoint
resource "azurerm_private_endpoint" "this" {
name = "kopicloud-openai-pe"
location = azurerm_resource_group.this.location
resource_group_name = azurerm_resource_group.this.name
subnet_id = azurerm_subnet.this.id
private_service_connection {
name = "kopicloud-openai-pe-psc"
private_connection_resource_id = azurerm_cognitive_account.this.id
subresource_names = ["account"]
is_manual_connection = false
}
private_dns_zone_group {
name = "default"
private_dns_zone_ids = [data.azurerm_private_dns_zone.openai_dns_zone.id]
}
}
9. Creating Azure OpenAI Variables
We will create some variables to deploy Azure Cognitive Services and OpenAI in the variables.tf file:
// Azure Cognitive Services Variables
variable "cognitive_account_sku_name" {
type = string
description = "Specifies the SKU Name for this Cognitive Service Account"
default = "S0"
}
variable "cognitive_account_kind_name" {
type = string
description = "Specifies the SKU Name for this Cognitive Service Account"
default = "OpenAI"
}
variable "cognitive_deployment_name" {
type = string
description = "The name of the Cognitive Services Account Deployment model"
default = "gpt-35-turbo"
}
variable "cognitive_deployment_version" {
type = string
description = "The version of the Cognitive Services Account Deployment model"
default = "0301"
}
variable "cognitive_deployment_scale_type" {
type = string
description = "Deployment scale type"
default = "Standard"
}
10. Deploying the Azure Cognitive Account
We will start the Azure OpenAI deployment with the Azure Cognitive Account instance:
resource "azurerm_cognitive_account" "this" {
name = "kopicloud-openai-aca"
location = azurerm_resource_group.this.location
resource_group_name = azurerm_resource_group.this.name
kind = var.cognitive_account_kind_name
sku_name = var.cognitive_account_sku_name
public_network_access_enabled = true
outbound_network_access_restricted = true
custom_subdomain_name = "kopicloud-openai"
network_acls {
default_action = "Deny"
virtual_network_rules {
subnet_id = azurerm_subnet.this.id
}
}
identity {
type = "SystemAssigned, UserAssigned"
identity_ids = [azurerm_user_assigned_identity.this.id]
}
}
Important: When we enable a private endpoint, it is important to configure the public access as in the example above, or the OpenAI Studio will not be able to communicate with our OpenAI instance.
Changing network rules can impact our applications’ ability to connect to Azure AI services. Setting the default network rule to deny blocks all access to the data unless specific network rules that grant access are also applied. Before we change the default rule to deny access, we need to grant access to any allowed networks by using network rules. If we allow listing for the IP addresses for our on-premises network, we need to add all possible outgoing public IP addresses from our on-premises network.
For more information, refer to the following Microsoft link.
11. Creating the Azure Cognitive Deployment
And the final step is to deploy our OpenAI GPT35 instance:
# Create the Azure Cognitive Deployments
resource "azurerm_cognitive_deployment" "this" {
name = "gpt35"
cognitive_account_id = azurerm_cognitive_account.this.id
model {
format = "OpenAI"
name = var.cognitive_deployment_name
version = var.cognitive_deployment_version
}
scale {
type = var.cognitive_deployment_scale_type
}
}
Note: the code deploys only one Azure Cognitive Deployment per Azure Cognitive Account, which is, for clarity, only. We will probably deploy multiple deployments per account. See the repo for a more complex example that deploys multiple deployments.
12. Troubleshooting and Errors
Below is a list of errors we got during our test of Azure OpenAI.
12.1. BringOwnFeatureNotEnabled: Bring your own feature is not enabled for Subscription/SKU/Kind
When trying to use the customer-managed key, you get “Unexpected status 400 with error: BringOwnFeatureNotEnabled: Bring your own feature is not enabled for Subscription/SKU/Kind” error:
module.openai.azurerm_cognitive_account_customer_managed_key.this: Creating…
╷
│ Error: adding Customer Managed Key for Account (
Subscription: "aaaaa-bbbb–cccc–dddd-ffffffffff"
│ Resource Group Name: "kopi-testai-rg"
│ Account Name: "kopi-testai-aca"): unexpected status 400 with error:
BringOwnFeatureNotEnabled: Bring your own feature is not enabled for
Subscription/SKU/Kind.
│
│ with module.openai.azurerm_cognitive_account_customer_managed_key.this,
│ on ..\terraform-azurerm-openai\openai_account.tf line 19, in resource
"azurerm_cognitive_account_customer_managed_key" "this":
│ 19: resource "azurerm_cognitive_account_customer_managed_key" "this" {
Solution:
This error means the customer-managed key feature is not enabled in your Azure subscription. Contact Microsoft or use one of the links above to request the feature.
12.2. InvalidResourceProperties: The specified SKU ‘Standard’ of account deployment is not supported in this region ‘westeurope’.
When I’m trying to deploy GPT4 in the West Europe region, I got this error:
Error: creating Deployment (Subscription: "aaaaa-bbbb–cccc–dddd-ffffffffff"
│ Resource Group Name: "kopi-testai-rg"
│ Account Name: "kopi-testai-aca"
│ Deployment Name: "kopi-testai-gpt4"): performing CreateOrUpdate:
unexpected status 400 with error: InvalidResourceProperties:
The specified SKU 'Standard' of account deployment is not supported in
this region 'westeurope'.
Solution:
At the moment of writing (October 2023), GPT-4 is available to customers in these regions:
- Australia East
- Canada East
- East US 2
- Japan East
- UK South
12.3. Error: Public access is disabled. Please configure private endpoint on Azure OpenAI Studio Chat Playground
When we try to use Azure OpenAI Chat from the Azure OpenAI Studio Chat Playground with a private endpoint enabled, we will receive the following message: Error: Public access is disabled. Please configure private endpoint.

Also, we can receive this error: Failed to retrieve fine tuned models — Public access is disabled. Please configure private endpoint.

Solution:
We must allow the Azure OpenAI to access the public network, restricting access only to our VNET and Subnet.
We open the Azure OpenAI service console and click on the Networking link.

Then click on the Selected Networks and Private Endpoints option.
- Click on Add existing virtual network, and select the VNET and subnet(s) where Azure OpenAI is configured.
- In the Firewall section, add the recommended external IP address.
Note: the Terraform code automatically takes care of these two steps above.

12.4. Error: Principal does not have access to API/Operation on Azure OpenAI Studio Chat Playground
When we try to use Azure OpenAI Chat from the Azure OpenAI Studio Chat Playground, we receive the error Principal does not have access to API/Operation:

Solution:
We need to assign the role “Cognitive Services OpenAI Contributor” or “Cognitive Services Contributor” to the user in the Azure portal or using Terraform code (see below).
Get the Object ID or Principal ID of the user on Microsoft Entra ID (Azure Portal):

Then use to assign to the “Cognitive Services OpenAI Contributor” role:
# Assign OpenAI Contributor Role to the Azure Portal User
resource "azurerm_role_assignment" "openai_contributor" {
scope = azurerm_cognitive_account.this.id
role_definition_name = "Cognitive Services OpenAI Contributor"
principal_id = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
Or the “Cognitive Services Contributor” role:
# Assign Cognitive Services Contributor Role to the Azure Portal User
resource "azurerm_role_assignment" "contributor" {
scope = azurerm_cognitive_account.this.id
role_definition_name = "Cognitive Services Contributor"
principal_id = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
Finally, log out from the Azure OpenAI Studio and log back.
12.5. Error: The Terraform Execution failed when Executing “azurerm_role_assignment”
When we run the Terraform code to assign permissions to a Principal ID or Object ID, we get the following error:
│ Error: authorization.RoleAssignmentsClient#Create: Failure responding to request: StatusCode=403 — Original Error: autorest/azure: Service returned an error. Status=403 Code=”AuthorizationFailed” Message=”The client ‘xxxx’ with object id ‘yyyyyyyyyyyyyyy’ does not have authorization to perform action ‘Microsoft.Authorization/roleAssignments/write’ over scope ‘/subscriptions/zzzzzzzzzzzz/resourceGroups/kopicloud-core-dev-we-rg/providers/Microsoft.CognitiveServices/accounts/kopicloud-core-dev-we-aca/providers/Microsoft.Authorization/roleAssignments/wwwwwwww’ or the scope is invalid. If access was recently granted, please refresh your credentials.” │ │ with azurerm_role_assignment.contributor, │ on openai-account.tf line 55, in resource “azurerm_role_assignment” “contributor”: │ 55: resource “azurerm_role_assignment” “contributor” {
Solution:
We need to add our Terraform service principal to the Owner role in the Azure portal.
The complete code is available at https://github.com/KopiCloud/terraform-azure-openai-private-endpoint
Also, check the code of a small .NET app I wrote to test the Azure OpenAI Chat GPT API at https://github.com/KopiCloud/dotnet-azure-openai-test
And that’s all, folks. If you liked this story, please show your support by 👏 this story. Thank you for reading!