Azure

This documentation will walk you through setting up Cloud API Adaptor (CAA) (a.k.a. Peer Pods) on Azure Kubernetes Service (AKS).

It explains how to deploy:

• A single worker node Kubernetes cluster using Azure Kubernetes Service (AKS),
• CAA on that Kubernetes cluster,
• A sample application deployed using CAA to verify that everything is working as expected.

Confidential Containers also supports using Azure Key Vault as a resource backend for Trustee. More info

Pre-requisites

Install Required Tools:

• Install kubectl,
• Install Helm,
• Install az CLI tool,
• Ensure that the tools curl, git, jq and sipcalc are installed.

Azure Preparation

Azure login

1. Authenticate with Azure by running the following command and following the instructions in the command line:

   az login
   

   Note: If you have access to multiple Azure accounts, make sure to select the one you want to use for this deployment in the command line after running az login.

2. Export the Azure subscription ID into AZ_SUBSCR_ID using the subscription currently selected in the Azure CLI context:

   export AZ_SUBSCR_ID=$(az account show --query id --output tsv)
   

   Verify that the correct subscription is selected

   az account show --query '{name:name, id:id, tenantId:tenantId}' -o yaml
   

3. Set the AZURE_REGION environment variable to the region you want to use for this deployment. This region will be used to deploy both the AKS cluster and the peer pod VMs.

   • AMD SEV-SNP
   • Intel® TDX
   • Non-Confidential

   export AZURE_REGION="eastus"
   

   Note: We selected the eastus region as it not only offers AMD SEV-SNP machines but also has prebuilt pod VM images readily available.

   export AZURE_REGION="eastus"
   

   Note: We selected the eastus region as it not only offers Intel® TDX machines but also has prebuilt pod VM images readily available.

   Note: Currently Intel® TDX machines are available in a limited number of regions:
   (westeurope, westus, WestUS3, eastus, EastUS2EUAP, northcentralus), so make sure to select a region that supports them.

   export AZURE_REGION="eastus"
   

   Note: We have chose region eastus because it has prebuilt pod VM images readily available.

Resource group

Note: Skip this step if you already have a resource group you want to use. Please, export the resource group name in the AZURE_RESOURCE_GROUP environment variable.

Create an Azure resource group by running the following command:

1. Export the AZURE_RESOURCE_GROUP environment variable to a unique name for the resource group:

   export AZURE_RESOURCE_GROUP="caa-rg-$(date '+%Y%m%b%d%H%M%S')"
   

2. Create the resource group in the specified region:

   az group create \
     --name "${AZURE_RESOURCE_GROUP}" \
     --location "${AZURE_REGION}"
   

Deploy Kubernetes using AKS

1. Export environment variables related to the AKS cluster deployment:

   export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
   export AKS_WORKER_USER_NAME="azuser"
   export AKS_RG="${AZURE_RESOURCE_GROUP}-aks"
   
   # SSH key is optional, but it allows user to SSH into the pod VMs for troubleshooting purposes. 
   # This option works only for custom debug enabled pod VM images. 
   # The prebuilt pod VM images do not have SSH connection enabled.
   export SSH_KEY=~/.ssh/id_rsa.pub
   

2. Deploy AKS with single worker node to the same resource group you have created:

   az aks create \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --node-resource-group "${AKS_RG}" \
     --name "${CLUSTER_NAME}" \
     --enable-oidc-issuer \
     --enable-workload-identity \
     --location "${AZURE_REGION}" \
     --node-count 1 \
     --node-vm-size Standard_F4s_v2 \
     --nodepool-labels node.kubernetes.io/worker= \
     --ssh-access disabled \
     --admin-username "${AKS_WORKER_USER_NAME}" \
     --os-sku Ubuntu
   

   Note: Optionally, deploy the worker nodes into an existing Azure Virtual Network (VNet) and subnet by adding the following flag: --vnet-subnet-id <MY_SUBNET_ID>.

   Note: For sample usage Standard_F4s_v2 machine type is used for the worker node - this machine is General purpose - without enabled confidentiality.

3. Download kubeconfig locally to access the cluster using kubectl:

   az aks get-credentials \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --name "${CLUSTER_NAME}"
   

User assigned identity and federated credentials

CAA needs privileges to talk to Azure API. This privilege is granted to CAA by associating a workload identity to the CAA service account. This workload identity (a.k.a. user assigned identity) is given permissions to create VMs, fetch images and join networks in the next step.

Note: If you use an existing AKS cluster it might need to be configured to support workload identity and OpenID Connect (OIDC), please refer to the instructions in this guide.

Create an identity for CAA:

1. Set the AZURE_WORKLOAD_IDENTITY_NAME environment variable to a name for the user assigned identity:

   export AZURE_WORKLOAD_IDENTITY_NAME="${CLUSTER_NAME}-identity"
   

2. Create the user assigned identity in Azure:

   az identity create \
     --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --location "${AZURE_REGION}"
   

3. Export the client ID of the user assigned identity to the environment variable USER_ASSIGNED_CLIENT_ID:

   export USER_ASSIGNED_CLIENT_ID="$(az identity show \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --query 'clientId' \
     -o tsv)"
   

Networking

The VMs that will host Pods will commonly require access to internet services, e.g. to pull images from a public OCI registry. Create a discrete subnet next to the AKS cluster subnet in the same VNet for peer pod VMs. Then create a NAT gateway, attach a public IP, and associate the NAT gateway with the peer-pods subnet to provide outbound connectivity.

1. Get the VNet name of the AKS cluster:

   export AZURE_VNET_NAME="$(az network vnet list \
                               -g ${AKS_RG} \
                               --query '[].name' \
                               -o tsv)"
   

2. Define the AKS subnet name.

   By default, the instructions assume aks-subnet, but this can differ depending on how the cluster was created.

   export AKS_SUBNET_NAME="aks-subnet"
   

   Show how to find subnet names in the VNet

   az network vnet subnet list \
     -g "${AKS_RG}" \
     --vnet-name "${AZURE_VNET_NAME}" \
     --query '[].name' \
     -o tsv
   

3. Get the CIDR of the AKS cluster subnet:

   export AKS_CIDR="$(az network vnet show \
     -n $AZURE_VNET_NAME \
     -g $AKS_RG \
     --query "subnets[?name == '${AKS_SUBNET_NAME}'].addressPrefix" \
     -o tsv)"
   

   Expected output example: 10.224.0.0/16.

4. Calculate the CIDR mask for the peer pod subnet by extracting the mask from the AKS CIDR:

   export MASK="${AKS_CIDR#*/}"
   

   Expected output example: 16.

5. Create a CIDR for the peer pod subnet by using sipcalc to calculate a new subnet from the AKS CIDR:

   PEERPOD_CIDR="$(
     sipcalc $AKS_CIDR -n 2 \
     | grep ^Network \
     | grep -v current \
     | cut -d' ' -f2
   )/${MASK}"
   

   Expected output example: 10.225.0.0/16.

6. Create a public IP for the NAT gateway:

   az network public-ip create -g "$AKS_RG" -n peerpod
   

7. Create a NAT gateway and attach the public IP to it:

   az network nat gateway create \
     -g "$AKS_RG" \
     -l "$AZURE_REGION" \
     --public-ip-addresses peerpod \
     -n peerpod
   

8. Create a subnet for the peer pods and attach the NAT gateway to it:

   az network vnet subnet create -g "$AKS_RG" \
     --vnet-name "$AZURE_VNET_NAME" \
     --nat-gateway peerpod \
     --address-prefixes "$PEERPOD_CIDR" \
     -n peerpod
   

9. Export the subnet ID to the environment variable AZURE_SUBNET_ID:

   export AZURE_SUBNET_ID="$(
     az network vnet subnet show \
     -g "$AKS_RG" \
     --vnet-name "$AZURE_VNET_NAME" \
     -n peerpod \
     --query id \
     -o tsv)"
   

AKS resource group permissions

For CAA to be able to manage VMs, assign Virtual Machine and Network related roles to the user assigned identity. The roles grant permissions to create VMs in AZURE_RESOURCE_GROUP and attach them to the VNet/subnet in AKS_RG.

1. Assign the “Virtual Machine Contributor” role to the user assigned identity:

   az role assignment create \
     --role "Virtual Machine Contributor" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"
   

2. Assign the “Reader” role to the user assigned identity:

   az role assignment create \
     --role "Reader" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"
   

3. Assign the “Network Contributor” role to the user assigned identity:

   az role assignment create \
     --role "Network Contributor" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AKS_RG}"
   

Create the federated credential for the CAA ServiceAccount using the OIDC endpoint from the AKS cluster:

1. Set the AKS_OIDC_ISSUER environment variable to the OIDC issuer URL of your AKS cluster:

   export AKS_OIDC_ISSUER="$(az aks show \
     --name "${CLUSTER_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --query "oidcIssuerProfile.issuerUrl" \
     -o tsv)"
   

2. Create the federated credential:

   az identity federated-credential create \
     --name "${CLUSTER_NAME}-federated" \
     --identity-name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --issuer "${AKS_OIDC_ISSUER}" \
     --subject system:serviceaccount:confidential-containers-system:cloud-api-adaptor \
     --audience api://AzureADTokenExchange
   

Deploy the CAA Helm chart

Note: If you are using Calico Container Network Interface (CNI) on the Kubernetes cluster, then, configure Virtual Extensible LAN (VXLAN) encapsulation for all inter workload traffic.

Download the CAA Helm chart

Currently, there are three options to choose from when downloading the CAA Helm chart:

• using the latest release,
• using the latest build from the main branch,
• using your own helm build.

• Versions:
• Last Release
• Latest Build
• DIY

CAA_VERSION="$(
  curl -fsSL \
    "https://api.github.com/repos/confidential-containers/cloud-api-adaptor/releases/latest" |
    jq -er '.tag_name | sub("^v"; "")'
)"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Azure.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export AZURE_IMAGE_ID="/CommunityGalleries/cococommunity-42d8482d-92cd-415b-b332-7648bd978eff/Images/peerpod-podvm-fedora/Versions/${CAA_VERSION}"

SUCCESS_TIME=$(curl -s \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/confidential-containers/cloud-api-adaptor/actions/workflows/azure-nightly-build.yml/runs?status=success" \
  | jq -r '.workflow_runs[0].updated_at')

export AZURE_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/$(date -u -jf "%Y-%m-%dT%H:%M:%SZ" "$SUCCESS_TIME" "+%Y.%m.%d" 2>/dev/null || date -d "$SUCCESS_TIME" +%Y.%m.%d)"

Set the CAA container image and tag

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

export CAA_TAG=""

Select peer-pods machine type

• AMD SEV-SNP
• Intel® TDX
• Non-Confidential

export AZURE_INSTANCE_SIZE="Standard_DC2as_v5"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_DC2es_v6"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_D2as_v5"
export DISABLECVM="true"

Populate the provider file

List of all available configuration options can be found in two places:

• Main charts values
• Azure specific values

Run the following command to update the providers/azure.yaml file:

cat <<EOF > providers/azure.yaml
provider: azure
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   azure:
      AZURE_IMAGE_ID: "${AZURE_IMAGE_ID}"
      AZURE_REGION: "${AZURE_REGION}"
      AZURE_RESOURCE_GROUP: "${AZURE_RESOURCE_GROUP}"
      AZURE_SUBNET_ID: "${AZURE_SUBNET_ID}"
      AZ_SUBSCR_ID: "${AZ_SUBSCR_ID}"
      AZURE_INSTANCE_SIZE: "${AZURE_INSTANCE_SIZE}"
      DISABLECVM: ${DISABLECVM}
EOF

Deploy helm chart

1. Create file namespace.yaml with the following content:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: confidential-containers-system
     labels:
       app.kubernetes.io/managed-by: Helm
     annotations:
       meta.helm.sh/release-name: peerpods
       meta.helm.sh/release-namespace: confidential-containers-system
   

   This namespace will be used to deploy CAA and related components, and it is labeled and annotated to be managed by Helm.

2. Create namespace managed by Helm:

   kubectl apply -f namespace.yaml
   

3. Create a Kubernetes Secret that stores the Azure service-account credentials:

   See providers/azure-secrets.yaml.template for required keys.

   Note: Below example assumes that you are using workload identity for authentication hence AZURE_CLIENT_SECRET and AZURE_TENANT_ID are not provided.

   kubectl create secret generic my-provider-creds \
     -n confidential-containers-system \
     --from-literal=AZURE_CLIENT_ID="${USER_ASSIGNED_CLIENT_ID}" \
     --from-file=id_rsa.pub=${SSH_KEY}
   

   Note: --from-file=id_rsa.pub=${SSH_KEY} is optional. It allows user to SSH into the pod VMs for troubleshooting purposes. This option works only for custom debug enabled pod VM images. The prebuilt pod VM images do not have SSH connection enabled.

4. Install helm chart:

   Below command uses customization options -f and --set which are described here.

   helm install peerpods . \
     -f providers/azure.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' \
     --dependency-update \
     -n confidential-containers-system
   

   Note: Above example assumes that you are using workload identity for authentication.
   This line: --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' is required only when using workload identity.

Generic Peer pods Helm charts deployment instructions are also described here.

Verify deployment

Verify that the runtimeclass is created after deploying Peer Pods Helm Charts:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Run sample application

To verify that everything is working as expected, you can deploy a sample application using the kata-remote runtime class provided by CAA.

1. Create a deployment file nginx-deployment.yaml for a sample nginx application:

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: nginx
     namespace: default
   spec:
     selector:
       matchLabels:
         app: nginx
     replicas: 1
     template:
       metadata:
         labels:
           app: nginx
       spec:
         runtimeClassName: kata-remote
         containers:
         - name: nginx
           image: nginx
           ports:
           - containerPort: 80
           imagePullPolicy: Always
   

2. Apply created configuration to the cluster

   kubectl apply -f nginx-deployment.yaml
   

3. Ensure that the pod is up and running:

   kubectl get pods -n default
   

4. Verify that the PodVM was created by running the following command:

   az vm list \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --output table
   

   In command output there should be the VM associated with the pod nginx.

Note: If you run into problems then check the troubleshooting guide here.

Pod VM reference values

Reference values for the PCRs can be retrieved from the registry and used in attestation policies to verify the integrity of the Pod VM image before trusting it with sensitive workloads or secrets.

As part of a Pod VM image build expected PCR measurements are published into an OCI registry.

Pre-requisites

Install ORAS tool to pull the reference values from the OCI registry and GitHub CLI to verify its build provenance.

Verify build provenance

Assert that the measurements have been generated by a trusted build process on the official repository. Specify --format=json to get more details about the build process.

CAA_REPO="confidential-containers/cloud-api-adaptor"
OCI_REGISTRY="ghcr.io/${CAA_REPO}/measurements/azure/podvm:${CAA_VERSION}"
gh attestation verify -R "$CAA_REPO" "oci://${OCI_REGISTRY}"

Retrieve reference values

The PCR values can be used in remote attestation policies to assert the integrity of the PodVM image.

$ oras pull "$OCI_REGISTRY"
$ jq -r .measurements.sha256.pcr11 < measurements.json
0x58e8afdf5b105fc6b202eb8e537a9f1512a4b33cd5921171b518645a86ca5a75

Uninstall

To uninstall Confidential Containers from Azure AKS cluster, use the following commands:

1. Remove all pods with kata-runtime runtime class:

   kubectl get pods -A -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,RUNTIMECLASS:.spec.runtimeClassName' \
   | grep kata-remote \
   | awk '{print $1, $2}' \
   | xargs -n 2 sh -c 'kubectl delete pod -n "$2" "$1"' _
   

2. List deployed Confidential Containers Helm chart:

   Note: This command assumes that only one Helm release is deployed in the confidential-containers-system namespace. If there are multiple releases, you may need to adjust the command to select the correct one.

   export HELM_COCO_CHART_NAME=$(helm list \
                                  -n confidential-containers-system \
                                  --short)
   

3. Delete Confidential Containers related Helm chart:

   helm uninstall ${HELM_COCO_CHART_NAME} \
     --namespace confidential-containers-system
   

4. Delete secret with provider credentials my-provider-creds:

   kubectl delete secret my-provider-creds \
     -n confidential-containers-system
   

5. Delete Confidential Containers related namespace:

   kubectl delete namespace confidential-containers-system
   

6. Delete the AKS cluster by running the following command:

   az group delete \
     --name "${AZURE_RESOURCE_GROUP}" \
     --yes --no-wait