Azure Kubernetes Service (AKS)

This documentation will walk you through setting up Cloud API Adaptor (CAA) (a.k.a. Peer Pods) on Azure Kubernetes Service (AKS).

It explains how to deploy:

• A single worker node Kubernetes cluster using Azure Kubernetes Service (AKS),
• CAA on that Kubernetes cluster,
• A sample application deployed using CAA to verify that everything is working as expected.

Confidential Containers also supports using Azure Key Vault as a resource backend for Trustee. More info

Before you begin

Complete the common prerequisites, including the Azure-specific tools. Then return here to prepare Azure.

AKS Preparation

Azure login

1. Authenticate with Azure by running the following command and following the instructions in the command line:

   az login
   

   Note: If you have access to multiple Azure accounts, make sure to select the one you want to use for this deployment in the command line after running az login.

2. Export the Azure subscription ID into AZ_SUBSCR_ID using the subscription currently selected in the Azure CLI context:

   export AZ_SUBSCR_ID=$(az account show --query id --output tsv)
   

   Verify that the correct subscription is selected

   az account show --query '{name:name, id:id, tenantId:tenantId}' -o yaml
   

3. Set the AZURE_REGION environment variable to the region you want to use for this deployment. This region will be used to deploy both the AKS cluster and the peer pod VMs.

   • AMD SEV-SNP
   • Intel® TDX
   • Non-Confidential

   export AZURE_REGION="eastus"
   

   Note: We selected the eastus region as it not only offers AMD SEV-SNP machines but also has prebuilt pod VM images readily available.

   export AZURE_REGION="eastus"
   

   Note: We selected the eastus region as it not only offers Intel® TDX machines but also has prebuilt pod VM images readily available.

   Note: Currently Intel® TDX machines are available in a limited number of regions:
   (westeurope, westus, WestUS3, eastus, EastUS2EUAP, northcentralus), so make sure to select a region that supports them.

   export AZURE_REGION="eastus"
   

   Note: We have chose region eastus because it has prebuilt pod VM images readily available.

Resource group

Note: Skip this step if you already have a resource group you want to use. Please, export the resource group name in the AZURE_RESOURCE_GROUP environment variable.

Create an Azure resource group by running the following command:

1. Export the AZURE_RESOURCE_GROUP environment variable to a unique name for the resource group:

   export AZURE_RESOURCE_GROUP="caa-rg-$(date '+%Y%m%b%d%H%M%S')"
   

2. Create the resource group in the specified region:

   az group create \
     --name "${AZURE_RESOURCE_GROUP}" \
     --location "${AZURE_REGION}"
   

Setup cluster

1. Export environment variables related to the AKS cluster deployment:

   export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
   export AKS_WORKER_USER_NAME="azuser"
   export AKS_RG="${AZURE_RESOURCE_GROUP}-aks"
   
   # SSH key is optional, but it allows user to SSH into the pod VMs for troubleshooting purposes. 
   # This option works only for custom debug enabled pod VM images. 
   # The prebuilt pod VM images do not have SSH connection enabled.
   export SSH_KEY=~/.ssh/id_rsa.pub
   

2. Deploy AKS with single worker node to the same resource group you have created:

   az aks create \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --node-resource-group "${AKS_RG}" \
     --name "${CLUSTER_NAME}" \
     --enable-oidc-issuer \
     --enable-workload-identity \
     --location "${AZURE_REGION}" \
     --node-count 1 \
     --node-vm-size Standard_F4s_v2 \
     --nodepool-labels node.kubernetes.io/worker= \
     --ssh-access disabled \
     --admin-username "${AKS_WORKER_USER_NAME}" \
     --os-sku Ubuntu
   

   Note: Optionally, deploy the worker nodes into an existing Azure Virtual Network (VNet) and subnet by adding the following flag: --vnet-subnet-id <MY_SUBNET_ID>.

   Note: For sample usage Standard_F4s_v2 machine type is used for the worker node - this machine is General purpose - without enabled confidentiality.

3. Download kubeconfig locally to access the cluster using kubectl:

   az aks get-credentials \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --name "${CLUSTER_NAME}"
   

User assigned identity and federated credentials

CAA needs privileges to talk to Azure API. This privilege is granted to CAA by associating a workload identity to the CAA service account. This workload identity (a.k.a. user assigned identity) is given permissions to create VMs, fetch images and join networks in the next step.

Note: If you use an existing AKS cluster it might need to be configured to support workload identity and OpenID Connect (OIDC), please refer to the instructions in this guide.

Create an identity for CAA:

1. Set the AZURE_WORKLOAD_IDENTITY_NAME environment variable to a name for the user assigned identity:

   export AZURE_WORKLOAD_IDENTITY_NAME="${CLUSTER_NAME}-identity"
   

2. Create the user assigned identity in Azure:

   az identity create \
     --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --location "${AZURE_REGION}"
   

3. Export the client ID of the user assigned identity to the environment variable USER_ASSIGNED_CLIENT_ID:

   export USER_ASSIGNED_CLIENT_ID="$(az identity show \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --query 'clientId' \
     -o tsv)"
   

Networking

The VMs that will host Pods will commonly require access to internet services, e.g. to pull images from a public OCI registry. Create a discrete subnet next to the AKS cluster subnet in the same VNet for peer pod VMs. Then create a NAT gateway, attach a public IP, and associate the NAT gateway with the peer-pods subnet to provide outbound connectivity.

1. Get the VNet name of the AKS cluster:

   export AZURE_VNET_NAME="$(az network vnet list \
                               -g ${AKS_RG} \
                               --query '[].name' \
                               -o tsv)"
   

2. Define the AKS subnet name.

   By default, the instructions assume aks-subnet, but this can differ depending on how the cluster was created.

   export AKS_SUBNET_NAME="aks-subnet"
   

   Show how to find subnet names in the VNet

   az network vnet subnet list \
     -g "${AKS_RG}" \
     --vnet-name "${AZURE_VNET_NAME}" \
     --query '[].name' \
     -o tsv
   

3. Get the CIDR of the AKS cluster subnet:

   export AKS_CIDR="$(az network vnet show \
     -n $AZURE_VNET_NAME \
     -g $AKS_RG \
     --query "subnets[?name == '${AKS_SUBNET_NAME}'].addressPrefix" \
     -o tsv)"
   

   Expected output example: 10.224.0.0/16.

4. Calculate the CIDR mask for the peer pod subnet by extracting the mask from the AKS CIDR:

   export MASK="${AKS_CIDR#*/}"
   

   Expected output example: 16.

5. Create a CIDR for the peer pod subnet by using sipcalc to calculate a new subnet from the AKS CIDR:

   PEERPOD_CIDR="$(
     sipcalc $AKS_CIDR -n 2 \
     | grep ^Network \
     | grep -v current \
     | cut -d' ' -f2
   )/${MASK}"
   

   Expected output example: 10.225.0.0/16.

6. Create a public IP for the NAT gateway:

   az network public-ip create -g "$AKS_RG" -n peerpod
   

7. Create a NAT gateway and attach the public IP to it:

   az network nat gateway create \
     -g "$AKS_RG" \
     -l "$AZURE_REGION" \
     --public-ip-addresses peerpod \
     -n peerpod
   

8. Create a subnet for the peer pods and attach the NAT gateway to it:

   az network vnet subnet create -g "$AKS_RG" \
     --vnet-name "$AZURE_VNET_NAME" \
     --nat-gateway peerpod \
     --address-prefixes "$PEERPOD_CIDR" \
     -n peerpod
   

9. Export the subnet ID to the environment variable AZURE_SUBNET_ID:

   export AZURE_SUBNET_ID="$(
     az network vnet subnet show \
     -g "$AKS_RG" \
     --vnet-name "$AZURE_VNET_NAME" \
     -n peerpod \
     --query id \
     -o tsv)"
   

AKS resource group permissions

For CAA to be able to manage VMs, assign Virtual Machine and Network related roles to the user assigned identity. The roles grant permissions to create VMs in AZURE_RESOURCE_GROUP and attach them to the VNet/subnet in AKS_RG.

1. Assign the “Virtual Machine Contributor” role to the user assigned identity:

   az role assignment create \
     --role "Virtual Machine Contributor" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"
   

2. Assign the “Reader” role to the user assigned identity:

   az role assignment create \
     --role "Reader" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"
   

3. Assign the “Network Contributor” role to the user assigned identity:

   az role assignment create \
     --role "Network Contributor" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AKS_RG}"
   

Create the federated credential for the CAA ServiceAccount using the OIDC endpoint from the AKS cluster:

1. Set the AKS_OIDC_ISSUER environment variable to the OIDC issuer URL of your AKS cluster:

   export AKS_OIDC_ISSUER="$(az aks show \
     --name "${CLUSTER_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --query "oidcIssuerProfile.issuerUrl" \
     -o tsv)"
   

2. Create the federated credential:

   az identity federated-credential create \
     --name "${CLUSTER_NAME}-federated" \
     --identity-name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --issuer "${AKS_OIDC_ISSUER}" \
     --subject system:serviceaccount:confidential-containers-system:cloud-api-adaptor \
     --audience api://AzureADTokenExchange
   

Deploy

Preparation

Note: If you are using Calico Container Network Interface (CNI) on the Kubernetes cluster, then, configure Virtual Extensible LAN (VXLAN) encapsulation for all inter workload traffic.

Complete Prepare the CAA deployment. This downloads the Helm artifacts, selects the CAA image, installs cert-manager, and creates the installation namespace.

Note: Keep the resulting CAA_VERSION, CAA_IMAGE, and CAA_TAG values in your terminal, then return here.

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Azure.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export AZURE_IMAGE_ID="/CommunityGalleries/cococommunity-42d8482d-92cd-415b-b332-7648bd978eff/Images/peerpod-podvm-fedora/Versions/${CAA_VERSION}"

SUCCESS_TIME=$(curl -s \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/confidential-containers/cloud-api-adaptor/actions/workflows/azure-nightly-build.yml/runs?status=success" \
  | jq -r '.workflow_runs[0].updated_at')

export AZURE_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/$(date -u -jf "%Y-%m-%dT%H:%M:%SZ" "$SUCCESS_TIME" "+%Y.%m.%d" 2>/dev/null || date -d "$SUCCESS_TIME" +%Y.%m.%d)"

Select peer-pods machine type

• AMD SEV-SNP
• Intel® TDX
• Non-Confidential

export AZURE_INSTANCE_SIZE="Standard_DC2as_v5"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_DC2es_v6"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_D2as_v5"
export DISABLECVM="true"

Populate the provider file

List of all available configuration options can be found in two places:

• Main charts values
• Azure specific values

Run the following command to update the providers/azure.yaml file:

cat <<EOF > providers/azure.yaml
provider: azure
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   azure:
      AZURE_IMAGE_ID: "${AZURE_IMAGE_ID}"
      AZURE_REGION: "${AZURE_REGION}"
      AZURE_RESOURCE_GROUP: "${AZURE_RESOURCE_GROUP}"
      AZURE_SUBNET_ID: "${AZURE_SUBNET_ID}"
      AZ_SUBSCR_ID: "${AZ_SUBSCR_ID}"
      AZURE_INSTANCE_SIZE: "${AZURE_INSTANCE_SIZE}"
      DISABLECVM: ${DISABLECVM}
EOF

Deploy helm chart

1. Create a Kubernetes Secret that stores the Azure service-account credentials:

   See providers/azure-secrets.yaml.template for required keys.

   Note: Below example assumes that you are using workload identity for authentication hence AZURE_CLIENT_SECRET and AZURE_TENANT_ID are not provided.

   kubectl create secret generic my-provider-creds \
     -n confidential-containers-system \
     --from-literal=AZURE_CLIENT_ID="${USER_ASSIGNED_CLIENT_ID}" \
     --from-file=id_rsa.pub=${SSH_KEY}
   

   Note: --from-file=id_rsa.pub=${SSH_KEY} is optional. It allows user to SSH into the pod VMs for troubleshooting purposes. This option works only for custom debug enabled pod VM images. The prebuilt pod VM images do not have SSH connection enabled.

2. Install the Helm chart:

   The -f and --set customization options are described in advanced configuration.

   helm install peerpods . \
     -f providers/azure.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' \
     --dependency-update \
     -n confidential-containers-system
   

   Note: Above example assumes that you are using workload identity for authentication.
   This line: --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' is required only when using workload identity.

Generic Peer pods Helm charts deployment instructions are also described here.

Verify deployment

Follow Verify the installation and run a sample workload, then confirm that Azure created the PodVM:

az vm list \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --output table

The output should include the VM associated with the nginx pod.

Note: If you run into problems, see the troubleshooting guide.

Pod VM reference values

Reference values for the PCRs can be retrieved from the registry and used in attestation policies to verify the integrity of the Pod VM image before trusting it with sensitive workloads or secrets.

As part of a Pod VM image build expected PCR measurements are published into an OCI registry.

Pre-requisites

Install ORAS tool to pull the reference values from the OCI registry and GitHub CLI to verify its build provenance.

Verify build provenance

Assert that the measurements have been generated by a trusted build process on the official repository. Specify --format=json to get more details about the build process.

CAA_REPO="confidential-containers/cloud-api-adaptor"
OCI_REGISTRY="ghcr.io/${CAA_REPO}/measurements/azure/podvm:${CAA_VERSION}"
gh attestation verify -R "$CAA_REPO" "oci://${OCI_REGISTRY}"

Retrieve reference values

The PCR values can be used in remote attestation policies to assert the integrity of the PodVM image.

$ oras pull "$OCI_REGISTRY"
$ jq -r .measurements.sha256.pcr11 < measurements.json
0x58e8afdf5b105fc6b202eb8e537a9f1512a4b33cd5921171b518645a86ca5a75

Uninstall

First remove the workloads and uninstall CAA. Then delete the resource group, including the AKS cluster and the Azure resources created by this guide:

az group delete \
  --name "${AZURE_RESOURCE_GROUP}" \
  --yes --no-wait