Quickstart: Create a private Azure Kubernetes Service (AKS) Automatic cluster in a custom virtual network

Create a virtual network

Create a virtual network using the az network vnet create command. Create an API server subnet and cluster subnet using the az network vnet subnet create command.

When using a custom virtual network with AKS Automatic, you must create an API server subnet. AKS will delegate the subnet to Microsoft.ContainerService/managedClusterson your behalf, which grants the AKS service permissions to inject the API server pods and internal load balancer into that subnet. You can't use the subnet for any other workloads, but you can use it for multiple AKS clusters located in the same virtual network. The minimum supported API server subnet size is a /28.

az network vnet create --name ${VNET_NAME} \
--resource-group ${RG_NAME} \
--location ${LOCATION} \
--address-prefixes 172.19.0.0/16

az network vnet subnet create --resource-group ${RG_NAME} \
--vnet-name ${VNET_NAME} \
--name apiServerSubnet \
--delegations Microsoft.ContainerService/managedClusters \
--address-prefixes 172.19.0.0/28

az network vnet subnet create --resource-group ${RG_NAME} \
--vnet-name ${VNET_NAME} \
--name userNodeSubnet \
--address-prefixes 172.19.1.0/24

az network vnet subnet create --resource-group ${RG_NAME} \
--vnet-name ${VNET_NAME} \
--name managedSystemNodeSubnet \
--address-prefixes 172.19.0.64/26

Network security group rules

All traffic within the virtual network is allowed by default. But if you added Network Security Group (NSG) rules to restrict traffic between different subnets, ensure that the NSG security rules permit the following types of communication:

Create a managed identity and give it permissions on the virtual network

Create a managed identity using the az identity create command and retrieve the principal ID. Assign the Network Contributor role on virtual network to the managed identity using the az role assignment create command.

az identity create \
--resource-group ${RG_NAME} \
 --name ${IDENTITY_NAME} \
 --location ${LOCATION}

IDENTITY_PRINCIPAL_ID=$(az identity show --resource-group ${RG_NAME} --name ${IDENTITY_NAME} --query principalId -o tsv)

az role assignment create \
--scope "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RG_NAME}/providers/Microsoft.Network/virtualNetworks/${VNET_NAME}" \
--role "Network Contributor" \
--assignee-object-id "${IDENTITY_PRINCIPAL_ID}" \
--assignee-principal-type ServicePrincipal

Create a private AKS Automatic cluster in a custom virtual network

To create a private AKS Automatic cluster, use the az aks create command. Note the use of the --enable-private-cluster flag.

az aks create \
--resource-group ${RG_NAME} \
--name ${CLUSTER_NAME} \
--location ${LOCATION} \
--apiserver-subnet-id "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RG_NAME}/providers/Microsoft.Network/virtualNetworks/${VNET_NAME}/subnets/apiServerSubnet" \
--node-subnet-id "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RG_NAME}/providers/Microsoft.Network/virtualNetworks/${VNET_NAME}/subnets/userNodeSubnet" \
--system-node-subnet-id "/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RG_NAME}/providers/Microsoft.Network/virtualNetworks/${VNET_NAME}/subnets/managedSystemNodeSubnet" 
--assign-identity "/subscriptions/${SUBSCRIPTION_ID}/resourcegroups/${RG_NAME}/providers/Microsoft.ManagedIdentity/userAssignedIdentities/${IDENTITY_NAME}" \
--sku automatic \
--enable-private-cluster \
--no-ssh-key

After a few minutes, the command completes and returns JSON-formatted information about the cluster.

Connect to the cluster

When an AKS Automatic cluster is created as a private cluster, the API server endpoint has no public IP address. To manage the API server, for example via kubectl, you need to connect through a machine that has access to the cluster's Azure virtual network. There are several options for establishing network connectivity to the private cluster:

• Create a virtual machine in the same virtual network as the AKS Automatic cluster using the az vm create command with the --vnet-name flag.
• Use a virtual machine in a separate virtual network and set up virtual network peering.
• Use an Express Route or VPN connection.
• Use a private endpoint connection.

Creating a virtual machine in the same virtual network as the AKS cluster is the easiest option. ExpressRoute and VPNs add costs and require other networking complexity. Virtual network peering requires you to plan your network CIDR ranges to ensure there are no overlapping ranges. For more information, see Options for connecting to the private cluster.

To manage a Kubernetes cluster, use the Kubernetes command-line client, kubectl. kubectl is already installed if you use Azure Cloud Shell. To install kubectl locally, run the az aks install-cli command. AKS Automatic clusters are configured with Microsoft Entra ID for Kubernetes role-based access control (RBAC).

When you create a cluster using the Azure CLI, your user is assigned built-in roles for Azure Kubernetes Service RBAC Cluster Admin.

Configure kubectl to connect to your Kubernetes cluster using the az aks get-credentials command. This command downloads credentials and configures the Kubernetes CLI to use them.

az aks get-credentials --resource-group ${RG_NAME} --name ${CLUSTER_NAME}

Verify the connection to your cluster using the kubectl get command. This command returns a list of the cluster nodes.

kubectl get nodes

The following sample output shows how you're asked to sign in.

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AAAAAAAAA to authenticate.

After you sign in, the following sample output shows the managed system node pools. Make sure the node status is Ready.

NAME                           STATUS   ROLES    AGE   VERSION
aks-hostedpool-16652789-vms1   Ready    <none>   19m   v1.34.7
aks-hostedpool-16652789-vms2   Ready    <none>   19m   v1.34.7
aks-hostedpool-16652789-vms3   Ready    <none>   19m   v1.34.7
aks-system-surge-zq4d2         Ready    <none>   19m   v1.34.7

Create a virtual network

This Bicep file defines a virtual network.

@description('The location of the managed cluster resource.')
param location string = resourceGroup().location

@description('The name of the virtual network.')
param vnetName string = 'aksAutomaticVnet'

@description('The address prefix of the virtual network.')
param addressPrefix string = '172.19.0.0/16'

@description('The name of the API server subnet.')
param apiServerSubnetName string = 'apiServerSubnet'

@description('The subnet prefix of the API server subnet.')
param apiServerSubnetPrefix string = '172.19.0.0/28'

@description('The name of the user node subnet.')
param userNodeSubnetName string = 'userNodeSubnet'

@description('The subnet prefix of the user node subnet.')
param userNodeSubnetPrefix string = '172.19.1.0/24'

@description('The name of the system node subnet.')
param systemNodeSubnetName string = 'systemNodeSubnet'

@description('The subnet prefix of the system node subnet.')
param systemNodeSubnetPrefix string = '172.19.0.64/26'

// Virtual network with an API server subnet, a user node subnet, and a system node subnet
resource virtualNetwork 'Microsoft.Network/virtualNetworks@2023-09-01' = {
    name: vnetName
    location: location
    properties: {
        addressSpace: {
            addressPrefixes: [ addressPrefix ]
        }
        subnets: [
            {
                name: apiServerSubnetName
                properties: {
                    addressPrefix: apiServerSubnetPrefix
                }
            }
            {
                name: userNodeSubnetName
                properties: {
                    addressPrefix: userNodeSubnetPrefix
                }
            }
            {
                name: systemNodeSubnetName
                properties: {
                    addressPrefix: systemNodeSubnetPrefix
                }
            }
        ]
    }
}

output apiServerSubnetId string = resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, apiServerSubnetName)
output userNodeSubnetId string = resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, userNodeSubnetName)
output systemNodeSubnetId string = resourceId('Microsoft.Network/virtualNetworks/subnets', vnetName, systemNodeSubnetName)

Save the Bicep file virtualNetwork.bicep to your local computer.

Deploy the Bicep file using the Azure CLI.

az deployment group create --resource-group <resource-group> --template-file virtualNetwork.bicep

All traffic within the virtual network is allowed by default. But if you added Network Security Group (NSG) rules to restrict traffic between different subnets, ensure that the NSG security rules permit the following types of communication:

Create a managed identity

This Bicep file defines a user assigned managed identity.

param location string = resourceGroup().location
param uamiName string = 'aksAutomaticUAMI'

resource userAssignedManagedIdentity 'Microsoft.ManagedIdentity/userAssignedIdentities@2023-01-31' = {
  name: uamiName
  location: location
}

output uamiId string = userAssignedManagedIdentity.id
output uamiPrincipalId string = userAssignedManagedIdentity.properties.principalId
output uamiClientId string = userAssignedManagedIdentity.properties.clientId

Save the Bicep file uami.bicep to your local computer.

Deploy the Bicep file using the Azure CLI.

az deployment group create --resource-group <resource-group> --template-file uami.bicep

Assign the Network Contributor role over the virtual network

This Bicep file defines role assignments over the virtual network.

@description('The name of the virtual network.')
param vnetName string = 'aksAutomaticVnet'

@description('The principal ID of the user assigned managed identity.')
param uamiPrincipalId string

// Get a reference to the virtual network
resource virtualNetwork 'Microsoft.Network/virtualNetworks@2023-09-01' existing ={
  name: vnetName
}

// Assign the Network Contributor role to the user assigned managed identity on the virtual network
// '4d97b98b-1d4f-4787-a291-c67834d212e7' is the built-in Network Contributor role definition
// See: https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/networking#network-contributor
resource networkContributorRoleAssignmentToVirtualNetwork 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  name: guid(uamiPrincipalId, '4d97b98b-1d4f-4787-a291-c67834d212e7', resourceGroup().id, virtualNetwork.name)
  scope: virtualNetwork
  properties: {
      roleDefinitionId: resourceId('Microsoft.Authorization/roleDefinitions', '4d97b98b-1d4f-4787-a291-c67834d212e7')
      principalId: uamiPrincipalId
  }
}

Save the Bicep file roleAssignments.bicep to your local computer.

Deploy the Bicep file using the Azure CLI. You need to provide the user assigned identity principal ID.

az deployment group create --resource-group <resource-group> --template-file roleAssignments.bicep \
--parameters uamiPrincipalId=<user assigned identity prinicipal id>

Create a private AKS Automatic cluster in a custom virtual network

This Bicep file defines the AKS Automatic cluster.

@description('The name of the managed cluster resource.')
param clusterName string = 'aksPrivateAutomaticCluster'

@description('The location of the managed cluster resource.')
param location string = resourceGroup().location

@description('The resource ID of the API server subnet.')
param apiServerSubnetId string

@description('The resource ID of the user node subnet.')
param userNodeSubnetId string

@description('The resource ID of the system node subnet.')
param systemNodeSubnetId string

@description('The resource ID of the user assigned managed identity.')
param uamiId string

/// Create the private AKS Automatic cluster using the custom virtual network and user assigned managed identity
resource aks 'Microsoft.ContainerService/managedClusters@2024-03-02-preview' = {
  name: clusterName
  location: location  
  sku: {
    name: 'Automatic'
  }
  properties: {
    apiServerAccessProfile: {
        subnetId: apiServerSubnetId
        enablePrivateCluster: true
    }
    networkProfile: {
      outboundType: 'loadBalancer'
    }
    hostedSystemProfile: {
      systemNodeSubnetID: systemNodeSubnetId
      nodeSubnetID: userNodeSubnetId
    }
  }
  identity: {
    type: 'UserAssigned'
    userAssignedIdentities: {
      '${uamiId}': {}
    }
  }
}

Save the Bicep file aks.bicep to your local computer.

Deploy the Bicep file using the Azure CLI. You need to provide the API server subnet resource ID, the user node subnet resource ID, the system node subnet resource ID, and user assigned managed identity resource ID.

az deployment group create --resource-group <resource-group> --template-file aks.bicep \
--parameters apiServerSubnetId=<API server subnet resource id> \
--parameters nodeSubnetId=<user node subnet resource id> \
--parameters systemNodeSubnetId=<system node subnet resource id> \
--parameters uamiPrincipalId=<user assigned identity prinicipal id>

Connect to the cluster

When an AKS Automatic cluster is created as a private cluster, the API server endpoint has no public IP address. To manage the API server, for example via kubectl, you need to connect through a machine that has access to the cluster's Azure virtual network. There are several options for establishing network connectivity to the private cluster:

• Create a virtual machine in the same virtual network as the AKS Automatic cluster using the az vm create command with the --vnet-name flag.
• Use a virtual machine in a separate virtual network and set up virtual network peering.
• Use an Express Route or VPN connection.
• Use a private endpoint connection.

Creating a virtual machine in the same virtual network as the AKS cluster is the easiest option. Express Route and VPNs add costs and require more networking complexity. Virtual network peering requires you to plan your network CIDR ranges to ensure there are no overlapping ranges. For more information, see Options for connecting to the private cluster.

To manage a Kubernetes cluster, use the Kubernetes command-line client, kubectl. kubectl is already installed if you use Azure Cloud Shell. To install kubectl locally, run the az aks install-cli command. AKS Automatic clusters are configured with Microsoft Entra ID for Kubernetes role-based access control (RBAC).

Configure kubectl to connect to your Kubernetes cluster using the az aks get-credentials command. This command downloads credentials and configures the Kubernetes CLI to use them.

az aks get-credentials --resource-group <resource-group> --name <cluster-name>

Verify the connection to your cluster using the kubectl get command. This command returns a list of the cluster nodes.

kubectl get nodes

The following sample output shows how you're asked to sign in.

To sign in, use a web browser to open the page https://microsoft.com/devicelogin and enter the code AAAAAAAAA to authenticate.

After you sign in, the following sample output shows the managed system node pools. Make sure the node status is Ready.

NAME                           STATUS   ROLES    AGE   VERSION
aks-hostedpool-16652789-vms1   Ready    <none>   19m   v1.34.7
aks-hostedpool-16652789-vms2   Ready    <none>   19m   v1.34.7
aks-hostedpool-16652789-vms3   Ready    <none>   19m   v1.34.7
aks-system-surge-zq4d2         Ready    <none>   19m   v1.34.7