Examples

• 1: Alibaba Cloud
• 2: AWS
• 3: Azure
• 4: Container Launch with SNP Memory Encryption
• 5: GCP

1 - Alibaba Cloud

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on Alibaba Cloud Container Service for Kubernetes (ACK) and Alibaba Cloud Elastic Compute Service (ECS). It explains how to deploy:

• One worker for ACK Managed Cluster
• CAA on that Kubernetes cluster
• An Nginx pod backed by CAA pod VM on ECS

Note: Run the following commands from the following directory - src/cloud-api-adaptor

Note: Now Confidential Computing instances are only available in cn-beijing region within zone cn-beijing-i.

Note: Official document from Alibaba Cloud can be found here.

Prerequisites

Install Required Tools:

• Install aliyun CLI tool and configure credentials,
• Have an aliyun OSS storage with a bucket.

Create pod VM Image

Note: There is a pre-built Community Image (id:m-2ze1w9aj2aonwckv64cw) for version 0.13.0 in cn-beijing that you can use for testing.

If you want to build a pod VM image yourself, please follow the steps.

1. Create pod VM image.

   PODVM_DISTRO=alinux \
   CLOUD_PROVIDER=alibabacloud \
   IMAGE_URL=https://alinux3.oss-cn-hangzhou.aliyuncs.com/aliyun_3_x64_20G_nocloud_alibase_20250117.qcow2 \
   make podvm-builder podvm-binaries podvm-image
   

   The built image will be available in the root path of following newly built docker image: quay.io/confidential-containers/podvm-alibabacloud-alinux-amd64:<sha256> with name like podvm-*.qcow2. You need to export it from the container image.

2. Upload to OSS storage and create ECS Image.

   You will then need to upload the Pod VM image to OSS (Object Storage Service).

   export REGION_ID=<region-id>
   export IMAGE_FILE=<path-to-qcow2-file>
   export BUCKET=<OSS-bucket-name>
   export OBJECT=<object-name>
   
   aliyun oss cp ${IMAGE_FILE} oss://${BUCKET}/${OBJECT}
   

   Then, mark the image file as an ECS Image

   export IMAGE_NAME=$(basename ${IMAGE_FILE%.*})
   
   aliyun ecs ImportImage --ImageName ${IMAGE_NAME} \
       --region ${REGION_ID} --RegionId ${REGION_ID}
       --BootMode UEFI \
       --DiskDeviceMapping.1.OSSBucket ${BUCKET} --DiskDeviceMapping.1.OSSObject ${OBJECT} \
       --Features.NvmeSupport supported \
       --method POST --force
   
   export POD_IMAGE_ID=<ImageId>
   

Build CAA development image

If you want to build CAA DaemonSet image yourself:

export registry=<registry-address>
export RELEASE_BUILD=true
export CLOUD_PROVIDER=alibabacloud
make image

After that you should take note of the tag used for this image, we will use it later.

Deploy Kubernetes using ACK Managed Cluster

1. Create ACK Managed Cluster.

   export CONTAINER_CIDR=172.18.0.0/16
   export REGION_ID=cn-beijing
   export ZONES='["cn-beijing-i"]'
   
   aliyun cs CreateCluster --header "Content-Type=application/json" --body "
   {
     \"cluster_type\":\"ManagedKubernetes\",
     \"name\":\"caa\",
     \"region_id\":\"${REGION_ID}\",
     \"zone_ids\":${ZONES},
     \"enable_rrsa\":true,
     \"container_cidr\":\"${CONTAINER_CIDR}\",
     \"addons\":[
       {
         \"name\":\"flannel\"
       }
     ]
   }"
   
   export CLUSTER_ID=<cluster-id>
   export SECURITY_GROUP_ID=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r  ".security_group_id")
   

   Wait for the cluster to be created. Get the vSwitch id of the cluster.

   VSWITCH_IDS=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r  ".parameters.WorkerVSwitchIds" | sed 's/^/["/; s/$/"]/; s/,/","/g')
   

   Then add one worker node to the cluster.

   WORKER_NODE_COUNT=1
   WORKER_NODE_TYPE="[\"ecs.g8i.xlarge\",\"ecs.g7.xlarge\"]"
   aliyun cs POST /clusters/${CLUSTER_ID}/nodepools \
     --region ${REGION_ID} \
     --header "Content-Type=application/json;" \
     --body "
   {
     \"nodepool_info\": {
       \"name\":\"worker-node-pool\"
     },
     \"management\":{
       \"enable\":true
     },
     \"scaling_group\":{
       \"desired_size\":${WORKER_NODE_COUNT},
       \"image_type\":\"Ubuntu\",
       \"instance_types\":${WORKER_NODE_TYPE},
       \"system_disk_category\":\"cloud_essd\",
       \"vswitch_ids\":${VSWITCH_IDS},
       \"system_disk_size\":40,
       \"internet_charge_type\":\"PayByTraffic\",
       \"internet_max_bandwidth_out\":5,
       \"data_disks\":[
         {
           \"size\":120,
           \"category\":\"cloud_essd\",
           \"system_disk_bursting_enabled\":false
         }
       ]
     }
   }"
   
   NODE_POOL_ID=<node-pool-id>
   

2. Add Internet access for the cluster VPC

   export VPC_ID=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r ".vpc_id")
   export VSWITCH_ID=$(echo ${VSWITCH_IDS} | sed 's/[][]//g' | sed 's/"//g')
   aliyun vpc CreateNatGateway \
     --region ${REGION_ID} \
     --RegionId ${REGION_ID} \
     --VpcId ${VPC_ID} \
     --NatType Enhanced \
     --VSwitchId ${VSWITCH_ID} \
     --NetworkType internet
   
   export GATEWAY_ID="<NatGatewayId>"
   export SNAT_TABLE_ID="<SnatTableId>"
   
   # The band width of the public ip (Mbps)
   export BAND_WIDTH=5
   aliyun vpc AllocateEipAddress \
     --region ${REGION_ID} \
     --RegionId ${REGION_ID} \
     --Bandwidth ${BAND_WIDTH}
   
   export EIP_ID="<AllocationId>"
   export EIP_ADDRESS="<EipAddress>"
   
   aliyun vpc AssociateEipAddress \
     --region ${REGION_ID} \
     --RegionId ${REGION_ID} \
     --AllocationId ${EIP_ID} \
     --InstanceId ${GATEWAY_ID} \
     --InstanceType Nat
   
   aliyun vpc CreateSnatEntry \
     --region ${REGION_ID} \
     --RegionId ${REGION_ID} \
     --SnatTableId ${SNAT_TABLE_ID} \
     --SourceVSwitchId ${VSWITCH_ID} \
     --SnatIp ${EIP_ADDRESS}
   

3. Grant role permissions

   Give role permission to the cluster to allow the worker to create ECS instances.

   export ROLE_NAME=caa-alibaba
   export RRSA_ISSUER=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r ".rrsa_config.issuer" | cut -d',' -f1)
   export RRSA_ARN=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r ".rrsa_config.oidc_arn" | cut -d',' -f1)
   
   aliyun ram CreateRole --region ${REGION_ID} \
     --RoleName ${ROLE_NAME} \
     --AssumeRolePolicyDocument "
   {
     \"Version\": \"1\",
     \"Statement\": [
       {
         \"Action\": \"sts:AssumeRole\",
         \"Condition\": {
           \"StringEquals\": {
             \"oidc:aud\": [
               \"sts.aliyuncs.com\"
             ],
             \"oidc:iss\": [
               \"${RRSA_ISSUER}\"
             ],
             \"oidc:sub\": [
               \"system:serviceaccount:confidential-containers-system:cloud-api-adaptor\"
             ]
           }
         },
         \"Effect\": \"Allow\",
         \"Principal\": {
           \"Federated\": [
             \"${RRSA_ARN}\"
           ]
         }
       }
     ]
   }"
   
   export POLICY_NAME=caa-aliyun-policy
   aliyun ram CreatePolicy --region ${REGION_ID} \
     --PolicyName ${POLICY_NAME} \
     --PolicyDocument "
     {
       \"Version\": \"1\",
       \"Statement\": [
         {
           \"Effect\": \"Allow\",
           \"Action\": [
             \"ecs:RunInstances\",
             \"ecs:DeleteInstance\",
             \"ecs:DescribeInstanceAttribute\",
             \"ecs:CreateNetworkInterface\",
             \"ecs:DeleteNetworkInterface\",
             \"ecs:AttachNetworkInterface\",
             \"ecs:ModifyNetworkInterfaceAttribute\",
             \"ecs:DescribeNetworkInterfaceAttribute\"
           ],
           \"Resource\": \"*\"
         },
         {
           \"Effect\": \"Allow\",
           \"Action\": \"vpc:DescribeVSwitchAttributes\",
           \"Resource\": \"*\"
         },
         {
           \"Effect\": \"Allow\",
           \"Action\": [
             \"vpc:AllocateEipAddress\",
             \"vpc:ReleaseEipAddress\",
             \"vpc:AssociateEipAddress\",
             \"vpc:UnassociateEipAddress\",
             \"vpc:DescribeEipAddresses\",
             \"vpc:DescribeVSwitchAttributes\"
           ],
           \"Resource\": \"*\"
         }
       ]
     }
     "
   
   aliyun ram AttachPolicyToRole \
     --region ${REGION_ID} \
     --PolicyType Custom \
     --PolicyName ${POLICY_NAME} \
     --RoleName ${ROLE_NAME}
   
   ROLE_ARN=$(aliyun ram GetRole --region ${REGION_ID} --RoleName ${ROLE_NAME} | jq -r ".Role.Arn")
   

Deploy CAA

Create the credentials file

cat <<EOF > install/overlays/alibabacloud/alibabacloud-cred.env
# If the WorkerNode is on ACK, we use RRSA to authenticate
ALIBABA_CLOUD_ROLE_ARN=${ROLE_ARN}
ALIBABA_CLOUD_OIDC_PROVIDER_ARN=${RRSA_ARN}
ALIBABA_CLOUD_OIDC_TOKEN_FILE=/var/run/secrets/ack.alibabacloud.com/rrsa-tokens/token
EOF

Update the kustomization.yaml file

At a minimum you need to update the following values in kustomization.yaml.

• VSWITCH_ID: Use one of the values of ${VSWITCH_IDS}
• SECURITY_GROUP_IDS: We can reuse the security group id ${SECURITY_GROUP_ID} of ACK.
• IMAGEID: The ECS images ID, e.g. m-2ze1w9aj2aonwckv64cw in cn-beijing region.
• REGION: The region where Peer Pods run, e.g. cn-beijing.

Deploy CAA on the Kubernetes cluster

Label the cluster nodes with node.kubernetes.io/worker=

for NODE_NAME in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label node $NODE_NAME node.kubernetes.io/worker=
done

Run the following command to deploy CAA:

Note: We have a forked version of CoCo Operator for Alibaba Cloud. Specifically, we enabled containerd 1.7+ installation and mirrored images from quay.io on Alibaba Cloud to accelerate.

export COCO_OPERATOR_REPO="https://github.com/AliyunContainerService/coco-operator"
export COCO_OPERATOR_REF="main"
export RESOURCE_CTRL=false
export CLOUD_PROVIDER=alibabacloud
make deploy

Generic CAA deployment instructions are also described here.

Run sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying CAA:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

Create an nginx deployment:

echo '
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  runtimeClassName: kata-remote
  containers:
  - name: nginx
    image: registry.openanolis.cn/openanolis/nginx:1.14.1-8.6
' | kubectl apply -f -

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the peer-pod VM was created by running the following command:

aliyun ecs DescribeInstances --RegionId ${REGION_ID} --InstanceName 'podvm-*'

Here you should see the VM associated with the pod nginx. If you run into problems then check the troubleshooting guide here.

Attestation

TODO

Cleanup

Delete all running pods using the runtimeClass kata-remote.

Verify that all peer-pod VMs are deleted. You can use the following command to list all the peer-pod VMs (VMs having prefix podvm) and status:

aliyun ecs DescribeInstances --RegionId ${REGION_ID} --InstanceName 'podvm-*'

Delete the ACK cluster by running the following command:

aliyun cs DELETE /clusters/${CLUSTER_ID} --region ${REGION_ID} --keep_slb false --retain_all_resources false --header "Content-Type=application/json;" --body "{}"

2 - AWS

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on AWS Elastic Kubernetes Service (EKS). It explains how to deploy:

• A single worker node Kubernetes cluster using Elastic Kubernetes Service (EKS)
• CAA on that Kubernetes cluster
• An Nginx pod backed by CAA pod VM

Pre-requisites

Install Required Tools:

• Install kubectl,
• Install Helm,
• Install aws CLI tool,
• Install eksctl CLI tool,
• Ensure that the tools curl, git and jq are installed.

AWS Preparation

• Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or AWS_PROFILE) and AWS_REGION for AWS CLI access

• Set the region:

• AMD SEV-SNP
• Non-Confidential

export AWS_REGION="us-east-2"

export AWS_REGION="us-east-2"

Deploy Kubernetes using EKS

Make changes to the following environment variable as you see fit:

export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
export CLUSTER_NODE_TYPE="m5.xlarge"
export CLUSTER_NODE_FAMILY_TYPE="Ubuntu2204"
export SSH_KEY=~/.ssh/id_rsa.pub

Example EKS cluster creation using the default AWS VPC-CNI

eksctl create cluster --name "$CLUSTER_NAME" \
    --node-type "$CLUSTER_NODE_TYPE" \
    --node-ami-family "$CLUSTER_NODE_FAMILY_TYPE" \
    --nodes 1 \
    --nodes-min 0 \
    --nodes-max 2 \
    --node-private-networking \
    --kubeconfig "$CLUSTER_NAME"-kubeconfig

Wait for the cluster to be created.

Label the cluster nodes with node.kubernetes.io/worker=

for NODE_NAME in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label node $NODE_NAME node.kubernetes.io/worker=
done

Allow required network ports

EKS_VPC_ID=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
--query "cluster.resourcesVpcConfig.vpcId" \
--output text)
echo $EKS_VPC_ID

EKS_CLUSTER_SG=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
  --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" \
  --output text)
echo $EKS_CLUSTER_SG

EKS_VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids "$EKS_VPC_ID" \
--query 'Vpcs[0].CidrBlock' --output text)
echo $EKS_VPC_CIDR

# agent-protocol-forwarder port
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 15150 --cidr "$EKS_VPC_CIDR"

# vxlan port
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 9000 --cidr "$EKS_VPC_CIDR"
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol udp --port 9000 --cidr "$EKS_VPC_CIDR"

Note:

• Port 15150 is the default port for CAA to connect to the agent-protocol-forwarder running inside the pod VM.
• Port 9000 is the VXLAN port used by CAA. Ensure it doesn’t conflict with the VXLAN port used by the Kubernetes CNI.

Deploy the CAA Helm chart

Download the CAA Helm deployment artifacts

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_VERSION="0.17.0"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in AWS.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export PODVM_AMI_ID=$(aws ec2 describe-images \
    --filters Name=name,Values="podvm-fedora-amd64-${CAA_VERSION//./-}" \
    --query 'Images[*].[ImageId]' --output text)

echo $PODVM_AMI_ID

Export CAA container image path

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

export CAA_TAG=""

Select peer-pods machine type

• AMD SEV-SNP
• Non-Confidential

export PODVM_INSTANCE_TYPE="m6a.large"
export DISABLECVM="false"

export PODVM_INSTANCE_TYPE="t3.large"
export DISABLECVM="true"

Populate the providers/aws.yaml file

List of all available configuration options can be found in two places:

• Main charts values
• AWS specific values

Run the following command to update the providers/aws.yaml file:

cat <<EOF > providers/aws.yaml
provider: aws
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   aws:
      DISABLECVM: ${DISABLECVM}
      PODVM_AMI_ID: "${PODVM_AMI_ID}"
      PODVM_INSTANCE_TYPE: "${PODVM_INSTANCE_TYPE}"
      VXLAN_PORT: 9000
EOF

Deploy helm chart on the Kubernetes cluster

1. Create namespace managed by Helm:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
    name: confidential-containers-system
    labels:
      app.kubernetes.io/managed-by: Helm
    annotations:
      meta.helm.sh/release-name: peerpods
      meta.helm.sh/release-namespace: confidential-containers-system
   EOF
   

2. Create the secret using kubectl:

   See providers/aws-secrets.yaml.template for required keys.

   kubectl create secret generic my-provider-creds \
   -n confidential-containers-system \
   --from-literal=AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
   --from-literal=AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
   --from-file=id_rsa.pub=${SSH_KEY}
   

   Note: --from-file=id_rsa.pub=${SSH_KEY} is optional. It allows user to SSH into the pod VMs for troubleshooting purposes. This option works only for custom debug enabled pod VM images. The prebuilt pod VM images do not have SSH connection enabled.

3. Install helm chart:

   Below command uses customization options -f and --set which are described here.

   helm install peerpods . \
     -f providers/aws.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --dependency-update \
     -n confidential-containers-system
   

Generic Peer pods Helm charts deployment instructions are also described here.

Run sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying CAA:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

Create an nginx deployment:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always
EOF

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the peer pod VM was created by running the following command:

aws ec2 describe-instances --filters "Name=tag:Name,Values=podvm*" \
   --query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value | [0]]' --output table

Here you should see the VM associated with the pod nginx.

Note: If you run into problems then check the troubleshooting guide here.

Cleanup

Delete all running pods using the runtimeclass kata-remote. You can use the following command for the same:

kubectl get pods -A -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,RUNTIMECLASS:.spec.runtimeClassName' | grep kata-remote | awk '{print $1, $2}'

Verify that all peer-pod VMs are deleted. You can use the following command to list all the peer-pod VMs (VMs having prefix podvm) and status:

aws ec2 describe-instances --filters "Name=tag:Name,Values=podvm*" \
--query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value | [0], State.Name]' --output table

Delete the EKS cluster by running the following command:

eksctl delete cluster --name=$CLUSTER_NAME 

3 - Azure

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on Azure Kubernetes Service (AKS). It explains how to deploy:

• A single worker node Kubernetes cluster using Azure Kubernetes Service (AKS)
• CAA on that Kubernetes cluster
• An Nginx pod backed by CAA pod VM

Confidential Containers also supports using Azure Key Vault as a resource backend for Trustee. More info

Pre-requisites

Install Required Tools:

• Install kubectl,
• Install Helm,
• Install az CLI tool,
• Ensure that the tools curl, git, jq and sipcalc are installed.

Azure Preparation

Azure login

There are a bunch of steps that require you to be logged into your Azure account:

az login

Retrieve your subscription ID:

export AZURE_SUBSCRIPTION_ID=$(az account show --query id --output tsv)

Set the region:

• AMD SEV-SNP
• Intel TDX
• Non-Confidential

export AZURE_REGION="eastus"

export AZURE_REGION="eastus2"

export AZURE_REGION="eastus"

Resource group

Note: Skip this step if you already have a resource group you want to use. Please, export the resource group name in the AZURE_RESOURCE_GROUP environment variable.

Create an Azure resource group by running the following command:

export AZURE_RESOURCE_GROUP="caa-rg-$(date '+%Y%m%b%d%H%M%S')"

az group create \
  --name "${AZURE_RESOURCE_GROUP}" \
  --location "${AZURE_REGION}"

Deploy Kubernetes using AKS

Make changes to the following environment variable as you see fit:

export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
export AKS_WORKER_USER_NAME="azuser"
export AKS_RG="${AZURE_RESOURCE_GROUP}-aks"
export SSH_KEY=~/.ssh/id_rsa.pub

Note: Optionally, deploy the worker nodes into an existing Azure Virtual Network (VNet) and subnet by adding the following flag: --vnet-subnet-id $MY_SUBNET_ID.

Deploy AKS with single worker node to the same resource group you created earlier:

az aks create \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --node-resource-group "${AKS_RG}" \
  --name "${CLUSTER_NAME}" \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --location "${AZURE_REGION}" \
  --node-count 1 \
  --node-vm-size Standard_F4s_v2 \
  --nodepool-labels node.kubernetes.io/worker= \
  --ssh-access disabled \
  --admin-username "${AKS_WORKER_USER_NAME}" \
  --os-sku Ubuntu

Download kubeconfig locally to access the cluster using kubectl:

az aks get-credentials \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --name "${CLUSTER_NAME}"

User assigned identity and federated credentials

CAA needs privileges to talk to Azure API. This privilege is granted to CAA by associating a workload identity to the CAA service account. This workload identity (a.k.a. user assigned identity) is given permissions to create VMs, fetch images and join networks in the next step.

Note: If you use an existing AKS cluster it might need to be configured to support workload identity and OpenID Connect (OIDC), please refer to the instructions in this guide.

Start by creating an identity for CAA:

export AZURE_WORKLOAD_IDENTITY_NAME="${CLUSTER_NAME}-identity"

az identity create \
  --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --location "${AZURE_REGION}"

export USER_ASSIGNED_CLIENT_ID="$(az identity show \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
  --query 'clientId' \
  -otsv)"

Networking

The VMs that will host Pods will commonly require access to internet services, e.g. to pull images from a public OCI registry. A discrete subnet can be created next to the AKS cluster subnet in the same VNet. We then attach a NAT gateway with a public IP to that subnet:

export AZURE_VNET_NAME="$(az network vnet list -g ${AKS_RG} --query '[].name' -o tsv)"
export AKS_CIDR="$(az network vnet show -n $AZURE_VNET_NAME -g $AKS_RG --query "subnets[?name == 'aks-subnet'].addressPrefix" -o tsv)"
# 10.224.0.0/16
export MASK="${AKS_CIDR#*/}"
# 16
PEERPOD_CIDR="$(sipcalc $AKS_CIDR -n 2 | grep ^Network | grep -v current | cut -d' ' -f2)/${MASK}"
# 10.225.0.0/16
az network public-ip create -g "$AKS_RG" -n peerpod
az network nat gateway create -g "$AKS_RG" -l "$AZURE_REGION" --public-ip-addresses peerpod -n peerpod
az network vnet subnet create -g "$AKS_RG" --vnet-name "$AZURE_VNET_NAME" --nat-gateway peerpod --address-prefixes "$PEERPOD_CIDR" -n peerpod
export AZURE_SUBNET_ID="$(az network vnet subnet show -g "$AKS_RG" --vnet-name "$AZURE_VNET_NAME" -n peerpod --query id -o tsv)"

AKS resource group permissions

For CAA to be able to manage VMs assign the identity VM and Network contributor roles, privileges to spawn VMs in $AZURE_RESOURCE_GROUP and attach to a VNet in $AKS_RG.

az role assignment create \
  --role "Virtual Machine Contributor" \
  --assignee "$USER_ASSIGNED_CLIENT_ID" \
  --scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"

az role assignment create \
  --role "Reader" \
  --assignee "$USER_ASSIGNED_CLIENT_ID" \
  --scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"

az role assignment create \
  --role "Network Contributor" \
  --assignee "$USER_ASSIGNED_CLIENT_ID" \
  --scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${AKS_RG}"

Create the federated credential for the CAA ServiceAccount using the OIDC endpoint from the AKS cluster:

export AKS_OIDC_ISSUER="$(az aks show \
  --name "${CLUSTER_NAME}" \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --query "oidcIssuerProfile.issuerUrl" \
  -otsv)"

az identity federated-credential create \
  --name "${CLUSTER_NAME}-federated" \
  --identity-name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --issuer "${AKS_OIDC_ISSUER}" \
  --subject system:serviceaccount:confidential-containers-system:cloud-api-adaptor \
  --audience api://AzureADTokenExchange

Deploy the CAA Helm chart

Note: If you are using Calico Container Network Interface (CNI) on the Kubernetes cluster, then, configure Virtual Extensible LAN (VXLAN) encapsulation for all inter workload traffic.

Download the CAA Helm deployment artifacts

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_VERSION="0.17.0"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Azure.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export AZURE_IMAGE_ID="/CommunityGalleries/cococommunity-42d8482d-92cd-415b-b332-7648bd978eff/Images/peerpod-podvm-fedora/Versions/${CAA_VERSION}"

SUCCESS_TIME=$(curl -s \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/confidential-containers/cloud-api-adaptor/actions/workflows/azure-nightly-build.yml/runs?status=success" \
  | jq -r '.workflow_runs[0].updated_at')

export AZURE_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/$(date -u -jf "%Y-%m-%dT%H:%M:%SZ" "$SUCCESS_TIME" "+%Y.%m.%d" 2>/dev/null || date -d "$SUCCESS_TIME" +%Y.%m.%d)"

Export CAA container image path

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

export CAA_TAG=""

Select peer-pods machine type

• AMD SEV-SNP
• Intel TDX
• Non-Confidential

export AZURE_INSTANCE_SIZE="Standard_DC2as_v5"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_DC2es_v6"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_D2as_v5"
export DISABLECVM="true"

Populate the providers/azure.yaml file

List of all available configuration options can be found in two places:

• Main charts values
• Azure specific values

Run the following command to update the providers/azure.yaml file:

cat <<EOF > providers/azure.yaml
provider: azure
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   azure:
      AZURE_IMAGE_ID: "${AZURE_IMAGE_ID}"
      AZURE_REGION: "${AZURE_REGION}"
      AZURE_RESOURCE_GROUP: "${AZURE_RESOURCE_GROUP}"
      AZURE_SUBNET_ID: "${AZURE_SUBNET_ID}"
      AZURE_SUBSCRIPTION_ID: "${AZURE_SUBSCRIPTION_ID}"
      AZURE_INSTANCE_SIZE: "${AZURE_INSTANCE_SIZE}"
      DISABLECVM: ${DISABLECVM}
EOF

Deploy helm chart on the Kubernetes cluster

1. Create namespace managed by Helm:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
    name: confidential-containers-system
    labels:
      app.kubernetes.io/managed-by: Helm
    annotations:
      meta.helm.sh/release-name: peerpods
      meta.helm.sh/release-namespace: confidential-containers-system
   EOF
   

2. Create the secret using kubectl:

   See providers/azure-secrets.yaml.template for required keys.

   Note: Below example assumes that you are using workload identity for authentication hence AZURE_CLIENT_SECRET and AZURE_TENANT_ID are not provided.

   kubectl create secret generic my-provider-creds \
   -n confidential-containers-system \
   --from-literal=AZURE_CLIENT_ID="${USER_ASSIGNED_CLIENT_ID}" \
   --from-file=id_rsa.pub=${SSH_KEY}
   

   Note: --from-file=id_rsa.pub=${SSH_KEY} is optional. It allows user to SSH into the pod VMs for troubleshooting purposes. This option works only for custom debug enabled pod VM images. The prebuilt pod VM images do not have SSH connection enabled.

3. Install helm chart:

   Below command uses customization options -f and --set which are described here.

   helm install peerpods . \
     -f providers/azure.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' \
     --dependency-update \
     -n confidential-containers-system
   

   Note: Above example assumes that you are using workload identity for authentication.
   This line: --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' is required only when using workload identity.

Generic Peer pods Helm charts deployment instructions are also described here.

Run sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying Peer Pods Helm Charts:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

Create an nginx deployment:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always
EOF

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the peer pod VM was created by running the following command:

az vm list \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --output table

Here you should see the VM associated with the pod nginx.

Note: If you run into problems then check the troubleshooting guide here.

Pod VM reference values

As part of a Pod VM image build expected PCR measurements are published into an OCI registry.

Pre-requisites

Install ORAS tool to pull the reference values from the OCI registry and GitHub CLI to verify its build provenance.

Verify build provenance

Assert that the measurements have been generated by a trusted build process on the official repository. Specify --format=json to get more details about the build process.

CAA_REPO="confidential-containers/cloud-api-adaptor"
OCI_REGISTRY="ghcr.io/${CAA_REPO}/measurements/azure/podvm:${CAA_VERSION}"
gh attestation verify -R "$CAA_REPO" "oci://${OCI_REGISTRY}"

Retrieve reference values

The PCR values can be used in remote attestation policies to assert the integrity of the PodVM image.

oras pull "$OCI_REGISTRY"
jq -r .measurements.sha256.pcr11 < measurements.json
0x58e8afdf5b105fc6b202eb8e537a9f1512a4b33cd5921171b518645a86ca5a75

Cleanup

If you wish to clean up the whole set up, you can delete the resource group by running the following command:

az group delete \
  --name "${AZURE_RESOURCE_GROUP}" \
  --yes --no-wait

4 - Container Launch with SNP Memory Encryption

Container Launch With Memory Encryption

Launch a Confidential Service

To launch a container with SNP memory encryption, the SNP runtime class (kata-qemu-snp) must be specified. A base alpine docker container (Dockerfile) has been previously built for testing purposes. This image has also been prepared with SSH access and provisioned with a SSH public key for validation purposes.

Here is a sample service yaml specifying the SNP runtime class:

kind: Service
apiVersion: v1
metadata:
  name: "confidential-unencrypted"
spec:
  selector:
    app: "confidential-unencrypted"
  ports:
  - port: 22
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: "confidential-unencrypted"
spec:
  selector:
    matchLabels:
      app: "confidential-unencrypted"
  template:
    metadata:
      labels:
        app: "confidential-unencrypted"
       annotations:
        io.containerd.cri.runtime-handler: kata-qemu-snp
    spec:
      runtimeClassName: kata-qemu-snp
      containers:
      - name: "confidential-unencrypted"
        image: ghcr.io/kata-containers/test-images:unencrypted-nightly
        imagePullPolicy: Always

Save the contents of this yaml to a file called confidential-unencrypted.yaml.

Start the service:

kubectl apply -f confidential-unencrypted.yaml

Check for errors:

kubectl describe pod confidential-unencrypted

If there are no errors in the Events section, then the container has been successfully created with SNP memory encryption.

Validate SNP Memory Encryption

The container dmesg log can be parsed to indicate that SNP memory encryption is enabled and active. The container image defined in the yaml sample above was built with a predefined key that is authorized for SSH access.

Get the pod IP:

pod_ip=$(kubectl get pod -o wide | grep confidential-unencrypted | awk '{print $6;}')

Download and save the SSH private key and set the permissions.

wget https://github.com/kata-containers/kata-containers/raw/main/tests/integration/kubernetes/runtimeclass_workloads/confidential/unencrypted/ssh/unencrypted -O confidential-image-ssh-key

chmod 600 confidential-image-ssh-key

The following command will run a remote SSH command on the container to check if SNP memory encryption is active:

ssh -i confidential-image-ssh-key \
  -o "StrictHostKeyChecking no" \
  -t root@${pod_ip} \
  'dmesg | grep "Memory Encryption Features""'

If SNP is enabled and active, the output should return:

[    0.150045] Memory Encryption Features active: AMD SNP

5 - GCP

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on Google Kubernetes Engine (GKE). It explains how to deploy:

• A single worker node Kubernetes cluster using GKE
• CAA on that Kubernetes cluster
• A sample application backed by a CAA pod VM

Pre-requisites

Install Required Tools:

• Install kubectl,
• Install Helm,
• Install gcloud CLI tool.

Google Cloud Project:

• Ensure you have a Google Cloud project created,
• Note the Project ID (export it as GCP_PROJECT_ID).

GCP Preparation

Start by authenticating with Google and choosing your project:

export GCP_PROJECT_ID="YOUR_PROJECT_ID"
gcloud auth login
gcloud config set project ${GCP_PROJECT_ID}

Enable the necessary API:

gcloud services enable container.googleapis.com --project=${GCP_PROJECT_ID}

Create a service account with the necessary permissions:

gcloud iam service-accounts create peerpods \
  --description="Peerpods Service Account" \
  --display-name="Peerpods Service Account"

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:peerpods@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/compute.instanceAdmin.v1"

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:peerpods@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

Generate and save the credentials file:

gcloud iam service-accounts keys create \
  ~/.config/gcloud/peerpods_application_key.json \
  --iam-account=peerpods@${GCP_PROJECT_ID}.iam.gserviceaccount.com

export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/peerpods_application_key.json

Configure additional environment variables that will be used later.

Set the region:

export GCP_REGION="us-central1"

Set the PodVM instance type:

• AMD SEV-SNP
• Intel TDX
• Non-Confidential

export PODVM_INSTANCE_TYPE="n2d-standard-4"
export DISABLECVM=false
export GCP_CONFIDENTIAL_TYPE="SEV" # SEV or SEV_SNP
export GCP_DISK_TYPE="pd-standard"

export PODVM_INSTANCE_TYPE="c3-standard-4"
export DISABLECVM=false
export GCP_CONFIDENTIAL_TYPE="TDX"
export GCP_DISK_TYPE="pd-balanced"

export PODVM_INSTANCE_TYPE="e2-medium"
export DISABLECVM=true
export GCP_CONFIDENTIAL_TYPE=""
export GCP_DISK_TYPE="pd-standard"

Deploy Kubernetes Using GKE

Deploy a single node Kubernetes cluster using GKE:

gcloud container clusters create my-cluster \
  --zone ${GCP_REGION}-a \
  --machine-type "e2-standard-4" \
  --image-type UBUNTU_CONTAINERD \
  --num-nodes 1

Label the worker nodes:

kubectl get nodes --selector='!node-role.kubernetes.io/master' -o name | \
xargs -I{} kubectl label {} node.kubernetes.io/worker=

Configure VPC network

We need to make sure port 15150 is open under the default VPC network:

gcloud compute firewall-rules create allow-port-15150 \
    --project=${GCP_PROJECT_ID} \
    --network=default \
    --allow=tcp:15150

For production scenarios, it is advisable to restrict the source IP range to minimize security risks. For example, you can restrict the source range to a specific IP address or CIDR block:

gcloud compute firewall-rules create allow-port-15150-restricted \
   --project=${GCP_PROJECT_ID} \
   --network=default \
   --allow=tcp:15150 \
   --source-ranges=[YOUR_EXTERNAL_IP]

Deploy the CAA Helm chart

Download the CAA Helm deployment artifacts

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_VERSION="0.17.0"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Google Cloud.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export PODVM_IMAGE_ID="/projects/it-cloud-gcp-prod-osc-devel/global/images/fedora-mkosi-tee-amd-1-11-0"

Export CAA container image path

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

export CAA_TAG=""

Populate the providers/gcp.yaml file

List of all available configuration options can be found in two places:

• Main charts values
• GCP specific values

Run the following command to update the providers/gcp.yaml file:

cat <<EOF > providers/gcp.yaml
provider: gcp
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
  gcp:
    GCP_NETWORK: "global/networks/default"
    GCP_PROJECT_ID: "${GCP_PROJECT_ID}"
    GCP_ZONE: "${GCP_REGION}-a"
    GCP_MACHINE_TYPE: "${PODVM_INSTANCE_TYPE}"
    GCP_DISK_TYPE: "${GCP_DISK_TYPE}"
    PODVM_IMAGE_NAME: "${PODVM_IMAGE_ID}"
    GCP_CONFIDENTIAL_TYPE: "${GCP_CONFIDENTIAL_TYPE}"
    DISABLECVM: ${DISABLECVM}
EOF

Deploy helm chart on the Kubernetes cluster

1. Create namespace managed by Helm:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
    name: confidential-containers-system
    labels:
      app.kubernetes.io/managed-by: Helm
    annotations:
      meta.helm.sh/release-name: peerpods
      meta.helm.sh/release-namespace: confidential-containers-system
   EOF
   

2. Create the secret using kubectl:

   See providers/gcp-secrets.yaml.template for required keys.

   kubectl create secret generic my-provider-creds \
    -n confidential-containers-system \
    --from-file=GCP_CREDENTIALS="${GOOGLE_APPLICATION_CREDENTIALS}"
   

3. Install helm chart:

   Below command uses customization options -f and --set which are described here.

   helm install peerpods . \
     -f providers/gcp.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --dependency-update \
     -n confidential-containers-system
   

Generic Peer pods Helm charts deployment instructions are also described here.

Run a sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying Peer Pods Helm Charts:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

• CoCo Secret Retrieval
• Basic nginx

# initdata.toml
algorithm = "sha384"
version = "0.1.0"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]
url = 'http://127.0.0.1:8080'

[token_configs.kbs]
url = 'http://127.0.0.1:8080'
cert = """
-----BEGIN CERTIFICATE-----
MIIDljCCAn6gAwIBAgIUR/UNh13GFam4emgludtype/S9BIwDQYJKoZIhvcNAQEL
BQAwdTELMAkGA1UEBhMCQ04xETAPBgNVBAgMCFpoZWppYW5nMREwDwYDVQQHDAhI
YW5nemhvdTERMA8GA1UECgwIQUFTLVRFU1QxFDASBgNVBAsMC0RldmVsb3BtZW50
MRcwFQYDVQQDDA5BQVMtVEVTVC1IVFRQUzAeFw0yNDAzMTgwNzAzNTNaFw0yNTAz
MTgwNzAzNTNaMHUxCzAJBgNVBAYTAkNOMREwDwYDVQQIDAhaaGVqaWFuZzERMA8G
A1UEBwwISGFuZ3pob3UxETAPBgNVBAoMCEFBUy1URVNUMRQwEgYDVQQLDAtEZXZl
bG9wbWVudDEXMBUGA1UEAwwOQUFTLVRFU1QtSFRUUFMwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDfp1aBr6LiNRBlJUcDGcAbcUCPG6UzywtVIc8+comS
ay//gwz2AkDmFVvqwI4bdp/NUCwSC6ShHzxsrCEiagRKtA3af/ckM7hOkb4S6u/5
ewHHFcL6YOUp+NOH5/dSLrFHLjet0dt4LkyNBPe7mKAyCJXfiX3wb25wIBB0Tfa0
p5VoKzwWeDQBx7aX8TKbG6/FZIiOXGZdl24DGARiqE3XifX7DH9iVZ2V2RL9+3WY
05GETNFPKtcrNwTy8St8/HsWVxjAzGFzf75Lbys9Ff3JMDsg9zQzgcJJzYWisxlY
g3CmnbENP0eoHS4WjQlTUyY0mtnOwodo4Vdf8ZOkU4wJAgMBAAGjHjAcMBoGA1Ud
EQQTMBGCCWxvY2FsaG9zdIcEfwAAATANBgkqhkiG9w0BAQsFAAOCAQEAKW32spii
t2JB7C1IvYpJw5mQ5bhIlldE0iB5rwWvNbuDgPrgfTI4xiX5sumdHw+P2+GU9KXF
nWkFRZ9W/26xFrVgGIS/a07aI7xrlp0Oj+1uO91UhCL3HhME/0tPC6z1iaFeZp8Y
T1tLnafqiGiThFUgvg6PKt86enX60vGaTY7sslRlgbDr9sAi/NDSS7U1PviuC6yo
yJi7BDiRSx7KrMGLscQ+AKKo2RF1MLzlJMa1kIZfvKDBXFzRd61K5IjDRQ4HQhwX
DYEbQvoZIkUTc1gBUWDcAUS5ztbJg9LCb9WVtvUTqTP2lGuNymOvdsuXq+sAZh9b
M9QaC1mzQ/OStg==
-----END CERTIFICATE-----
"""
'''

"cdh.toml"  = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = 'http://1.2.3.4:8080'
kbs_cert = """
-----BEGIN CERTIFICATE-----
MIIFTDCCAvugAwIBAgIBADBGBgkqhkiG9w0BAQowOaAPMA0GCWCGSAFlAwQCAgUA
oRwwGgYJKoZIhvcNAQEIMA0GCWCGSAFlAwQCAgUAogMCATCjAwIBATB7MRQwEgYD
VQQLDAtFbmdpbmVlcmluZzELMAkGA1UEBhMCVVMxFDASBgNVBAcMC1NhbnRhIENs
YXJhMQswCQYDVQQIDAJDQTEfMB0GA1UECgwWQWR2YW5jZWQgTWljcm8gRGV2aWNl
czESMBAGA1UEAwwJU0VWLU1pbGFuMB4XDTIzMDEyNDE3NTgyNloXDTMwMDEyNDE3
NTgyNlowejEUMBIGA1UECwwLRW5naW5lZXJpbmcxCzAJBgNVBAYTAlVTMRQwEgYD
VQQHDAtTYW50YSBDbGFyYTELMAkGA1UECAwCQ0ExHzAdBgNVBAoMFkFkdmFuY2Vk
IE1pY3JvIERldmljZXMxETAPBgNVBAMMCFNFVi1WQ0VLMHYwEAYHKoZIzj0CAQYF
K4EEACIDYgAExmG1ZbuoAQK93USRyZQcsyobfbaAEoKEELf/jK39cOVJt1t4s83W
XM3rqIbS7qHUHQw/FGyOvdaEUs5+wwxpCWfDnmJMAQ+ctgZqgDEKh1NqlOuuKcKq
2YAWE5cTH7sHo4IBFjCCARIwEAYJKwYBBAGceAEBBAMCAQAwFwYJKwYBBAGceAEC
BAoWCE1pbGFuLUIwMBEGCisGAQQBnHgBAwEEAwIBAzARBgorBgEEAZx4AQMCBAMC
AQAwEQYKKwYBBAGceAEDBAQDAgEAMBEGCisGAQQBnHgBAwUEAwIBADARBgorBgEE
AZx4AQMGBAMCAQAwEQYKKwYBBAGceAEDBwQDAgEAMBEGCisGAQQBnHgBAwMEAwIB
CDARBgorBgEEAZx4AQMIBAMCAXMwTQYJKwYBBAGceAEEBEDDhCejDUx6+dlvehW5
cmmCWmTLdqI1L/1dGBFdia1HP46MC82aXZKGYSutSq37RCYgWjueT+qCMBE1oXDk
d1JOMEYGCSqGSIb3DQEBCjA5oA8wDQYJYIZIAWUDBAICBQChHDAaBgkqhkiG9w0B
AQgwDQYJYIZIAWUDBAICBQCiAwIBMKMDAgEBA4ICAQACgCai9x8DAWzX/2IelNWm
ituEBSiq9C9eDnBEckQYikAhPasfagnoWFAtKu/ZWTKHi+BMbhKwswBS8W0G1ywi
cUWGlzigI4tdxxf1YBJyCoTSNssSbKmIh5jemBfrvIBo1yEd+e56ZJMdhN8e+xWU
bvovUC2/7Dl76fzAaACLSorZUv5XPJwKXwEOHo7FIcREjoZn+fKjJTnmdXce0LD6
9RHr+r+ceyE79gmK31bI9DYiJoL4LeGdXZ3gMOVDR1OnDos5lOBcV+quJ6JujpgH
d9g3Sa7Du7pusD9Fdap98ocZslRfFjFi//2YdVM4MKbq6IwpYNB+2PCEKNC7SfbO
NgZYJuPZnM/wViES/cP7MZNJ1KUKBI9yh6TmlSsZZOclGJvrOsBZimTXpATjdNMt
cluKwqAUUzYQmU7bf2TMdOXyA9iH5wIpj1kWGE1VuFADTKILkTc6LzLzOWCofLxf
onhTtSDtzIv/uel547GZqq+rVRvmIieEuEvDETwuookfV6qu3D/9KuSr9xiznmEg
xynud/f525jppJMcD/ofbQxUZuGKvb3f3zy+aLxqidoX7gca2Xd9jyUy5Y/83+ZN
bz4PZx81UJzXVI9ABEh8/xilATh1ZxOePTBJjN7lgr0lXtKYjV/43yyxgUYrXNZS
oLSG2dLCK9mjjraPjau34Q==
-----END CERTIFICATE-----
"""
'''

"policy.rego" = '''
package agent_policy

import future.keywords.in
import future.keywords.every

import input

# Default values, returned by OPA when rules cannot be evaluated to true.
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := false
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default OnlineCPUMemRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StatsContainerRequest := true
default TtyWinResizeRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := false
'''

INITDATA=$(cat initdata.toml | gzip | base64 -w0)

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: "$INITDATA"
spec:
  runtimeClassName: kata-remote
  containers:
    - name: example-container
      image: alpine:latest
      command:
        - sleep
        - "3600"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
EOF

kubectl exec -it example-pod -- curl http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always
EOF

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the PodVM was created by running the following command:

gcloud compute instances list

Here you should see the VM associated with the pod used by the example above.

Cleanup

Delete all running pods using the runtimeclass kata-remote. You can use the following command for the same:

kubectl get pods -A -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,RUNTIMECLASS:.spec.runtimeClassName' | grep kata-remote | awk '{print $1, $2}'

Verify that all peer-pod VMs are deleted. You can use the following command to list all the peer-pod VMs (VMs having prefix podvm) and status:

gcloud compute instances list \
  --filter="name~'podvm.*'" \
  --format="table(name,zone,status)"

Delete the GKE cluster by running the following command:

gcloud container clusters delete my-cluster --zone ${GCP_REGION}-a

Troubleshooting

Note: If your case is not covered in section below check the troubleshooting guide here.

VM Doesn’t Start

Starting with GKE version 1.27, GCP configures containerd with the discard_unpacked_layers=true flag to optimize disk usage by removing compressed image layers after they are unpacked. However, this can cause issues with PeerPods, as the workload may fail to locate required layers. To avoid this, disable the discard_unpacked_layers setting in the containerd configuration.

Most of the time you will see a generic message such as the following:

Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:<SHA>: failed to get reader from content store: content digest sha256:<SHA>: not found

To disable the discard_unpacked_layers setting in the containerd configuration on Google Kubernetes Engine (GKE) version 1.27 or later, follow these steps:

1. SSH to worker node Google console
2. Run command sudo sed -i 's/discard_unpacked_layers = true/discard_unpacked_layers = false/' /etc/containerd/config.toml
3. Verify changed property sudo cat /etc/containerd/config.toml | grep discard_unpacked_layers
4. Restart containerd sudo systemctl restart containerd