Cloud Providers

• 1: Common Cloud API Adaptor setup
• 2: Providers
• 2.1: Alibaba Cloud (ACK)
• 2.2: AWS (EKS)
• 2.3: Azure Kubernetes Service (AKS)
• 2.4: Google Cloud Platform (GCP)
• 3: Verify and clean up a CAA deployment

1 - Common Cloud API Adaptor setup

Use this page together with one provider guide. The provider guide tells you when to complete each shared step and supplies the cloud-specific cluster, network, identity, PodVM image, and Helm values.

Prerequisites

Install the tools used by every provider guide:

• kubectl
• Helm
• curl, git, and jq

Install the additional tools for your provider:

Authenticate the provider CLI before creating cloud resources. Continue with your provider guide to prepare the cluster, networking, and identity.

Prepare the CAA deployment

Complete the following steps when the provider guide directs you to prepare the shared deployment files.

Download the Helm deployment artifacts

Choose one source. Each option leaves the terminal in the peerpods Helm chart directory.

• Versions:
• Last Release
• Latest Build
• Local Checkout

CAA_VERSION="$(
  curl -fsSL \
    "https://api.github.com/repos/confidential-containers/cloud-api-adaptor/releases/latest" |
    jq -er '.tag_name | sub("^v"; "")'
)"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

cd <cloud-api-adaptor-checkout>/src/cloud-api-adaptor/install/charts/peerpods

Select the CAA container image

Use an image that matches the source selected above.

• Versions:
• Last Release
• Latest Build
• Custom Image

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="<CI_IMAGE_TAG>"

export CAA_IMAGE="<REGISTRY>/<IMAGE>"
export CAA_TAG="<TAG>"

Install cert-manager

Note: Skip this step if cert-manager is already installed in the cluster.

The Peer Pods chart uses cert-manager for webhook certificates.

# Add the Jetstack Helm repository
helm repo add jetstack https://charts.jetstack.io
helm repo update

# Install cert-manager with CRDs
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.19.1 \
  --set crds.enabled=true

Wait for cert-manager to be ready:

kubectl wait --for=condition=ready pod \
  -l app.kubernetes.io/instance=cert-manager \
  -n cert-manager \ 
  --timeout=300s

Prepare the installation namespace

Create the namespace and add the metadata Helm needs to manage it:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Namespace
metadata:
  name: confidential-containers-system
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    meta.helm.sh/release-name: peerpods
    meta.helm.sh/release-namespace: confidential-containers-system
EOF

Return to your provider guide to configure the PodVM image, machine type, credentials, and provider values before installing the chart.

2 - Providers

2.1 - Alibaba Cloud (ACK)

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on Alibaba Cloud Container Service for Kubernetes (ACK) and Alibaba Cloud Elastic Compute Service (ECS). It explains how to deploy:

• One worker for ACK Managed Cluster
• CAA on that Kubernetes cluster
• An Nginx pod backed by CAA pod VM on ECS

Note: Run the following commands from the following directory - src/cloud-api-adaptor

Note: Now Confidential Computing instances are available in some regions.

Note: Official document from Alibaba Cloud can be found here.

Before you begin

Complete the common prerequisites, including the Alibaba Cloud CLI, and configure its credentials. To build a custom PodVM image, you also need an OSS bucket.

ACK Preparation

Setup cluster

1. Create ACK Managed Cluster.

  export CONTAINER_CIDR=172.18.0.0/16
  export REGION_ID=cn-beijing
  export ZONES='["cn-beijing-i"]'

  aliyun cs CreateCluster --header "Content-Type=application/json" --body "
  {
    \"cluster_type\":\"ManagedKubernetes\",
    \"name\":\"caa\",
    \"region_id\":\"${REGION_ID}\",
    \"zone_ids\":${ZONES},
    \"enable_rrsa\":true,
    \"container_cidr\":\"${CONTAINER_CIDR}\",
    \"addons\":[
      {
        \"name\":\"flannel\"
      }
    ]
  }"

  export CLUSTER_ID=<cluster-id>
  export SECURITY_GROUP_ID=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r  ".security_group_id")

Wait for the cluster to be created. Get the vSwitch id of the cluster. Then add one worker node to the cluster.

2. Add Internet access for the cluster VPC

  export VPC_ID=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r ".vpc_id")
  export VSWITCH_ID=$(echo ${VSWITCH_IDS} | sed 's/[][]//g' | sed 's/"//g')
  aliyun vpc CreateNatGateway \
    --region ${REGION_ID} \
    --RegionId ${REGION_ID} \
    --VpcId ${VPC_ID} \
    --NatType Enhanced \
    --VSwitchId ${VSWITCH_ID} \
    --NetworkType internet

  export GATEWAY_ID="<NatGatewayId>"
  export SNAT_TABLE_ID="<SnatTableId>"

  # The band width of the public ip (Mbps)
  export BAND_WIDTH=5
  aliyun vpc AllocateEipAddress \
    --region ${REGION_ID} \
    --RegionId ${REGION_ID} \
    --Bandwidth ${BAND_WIDTH}

  export EIP_ID="<AllocationId>"
  export EIP_ADDRESS="<EipAddress>"

  aliyun vpc AssociateEipAddress \
    --region ${REGION_ID} \
    --RegionId ${REGION_ID} \
    --AllocationId ${EIP_ID} \
    --InstanceId ${GATEWAY_ID} \
    --InstanceType Nat

  aliyun vpc CreateSnatEntry \
    --region ${REGION_ID} \
    --RegionId ${REGION_ID} \
    --SnatTableId ${SNAT_TABLE_ID} \
    --SourceVSwitchId ${VSWITCH_ID} \
    --SnatIp ${EIP_ADDRESS}

3. Grant role permissions Give role permission to the cluster to allow the worker to create ECS instances.

Create the PodVM image

Note: There is a pre-built Community Image (id:m-j6c3kfz2vhu6ze04wacb) for version 0.22.0 in cn-hongkong that you can use for testing. You can also use the region-specific image IDs listed under Export PodVM image version below.

If you want to build a pod VM image yourself, follow the steps below. The current PodVM build uses mkosi and targets Ubuntu 26.04 (resolute). See the PodVM README for prerequisites and customization options.

1. Build the Ubuntu 26.04 PodVM image.

cd podvm
make

This builds the PodVM binaries and the OS image. The resulting qcow2 file is:

podvm/build/podvm-ubuntu-amd64.qcow2

To rebuild only the OS image after binaries are already present:

cd podvm
make image

2. Upload to OSS storage and create an ECS Image.

From src/cloud-api-adaptor, upload the qcow2 file to OSS (Object Storage Service):

cd ..
export REGION_ID=<region-id>
export IMAGE_FILE=podvm/build/podvm-ubuntu-amd64.qcow2
export BUCKET=<OSS-bucket-name>
export OBJECT=<object-name>

aliyun oss cp ${IMAGE_FILE} oss://${BUCKET}/${OBJECT}

Then import it as an ECS image:

export IMAGE_NAME=$(basename ${IMAGE_FILE%.*})
aliyun ecs ImportImage --ImageName ${IMAGE_NAME} \
    --region ${REGION_ID} --RegionId ${REGION_ID} \
    --BootMode UEFI \
    --DiskDeviceMapping.1.OSSBucket ${BUCKET} --DiskDeviceMapping.1.OSSObject ${OBJECT} \
    --Features.NvmeSupport supported \
    --method POST --force

export POD_IMAGE_ID=<ImageId>

Deploy

Preparation

Complete Prepare the CAA deployment.

This downloads the Helm artifacts, selects the CAA image, installs cert-manager, and creates the installation namespace.

Note: Keep the resulting CAA_VERSION, CAA_IMAGE, and CAA_TAG values in your terminal, then return here.

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Alibaba Cloud.

export IMAGEID="m-j6c3kfz2vhu6ze04wacb"

Note: Alibaba Cloud builds the images ahead of time. Different regions has different image id to use.

region	IMAGEID
cn-beijing	m-2ze2vxvrxsbue3sf8b02
cn-hongkong	m-j6c3kfz2vhu6ze04wacb
cn-hangzhou	m-bp146ws6x3iyuwjvbdw2
ap-southeast-1	m-t4ng1w8ipua3c0o57hor

Select peer-pods machine type

export PODVM_INSTANCE_TYPE="ecs.g8i.xlarge"
export DISABLECVM="false"

Note: See the official document for more instance types that supports confidential computing.

Populate the provider file

List of all available configuration options can be found in two places:

• Main charts values
• Alibaba Cloud specific values

Run the following command to update the providers/alibabacloud.yaml file:

cat <<EOF > providers/alibabacloud.yaml
provider: alibabacloud
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   alibabacloud:
      IMAGEID: "${IMAGEID}"
      REGION: "${REGION_ID}"
      SECURITY_GROUP_IDS: "${SECURITY_GROUP_ID}"
      VSWITCH_ID: "${VSWITCH_ID}"
      DISABLECVM: ${DISABLECVM}
alibabacloud:
  rrsa:
    enable: true
EOF

Note: If you are not using RRSA for auth, please change the alibabacloud.rrsa.enable to false in the yaml.

Deploy helm chart

1. Create the secret using kubectl:

   See providers/alibabacloud-secrets.yaml.template for required keys.

   Note: Below example assumes that you are using RRSA for auth hence ALIBABACLOUD_ACCESS_KEY_ID and ALIBABACLOUD_ACCESS_KEY_SECRET are not provided, while ALIBABA_CLOUD_ROLE_ARN and ALIBABA_CLOUD_OIDC_PROVIDER_ARN are provided.

   kubectl create secret generic my-provider-creds \
   -n confidential-containers-system \
   --from-literal=ALIBABA_CLOUD_ROLE_ARN=${ALIBABA_CLOUD_ROLE_ARN} \
   --from-literal=ALIBABA_CLOUD_OIDC_PROVIDER_ARN=${ALIBABA_CLOUD_OIDC_PROVIDER_ARN} \
   --from-literal=ALIBABA_CLOUD_OIDC_TOKEN_FILE=/var/run/secrets/ack.alibabacloud.com/rrsa-tokens/token
   

2. Install the Helm chart:

   The -f and --set customization options are described in advanced configuration.

   helm install peerpods . \
     -f providers/alibabacloud.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --dependency-update \
     -n confidential-containers-system
   

Generic Peer pods Helm charts deployment instructions are also described here.

Verify deployment

Follow Verify the installation and run a sample workload, then confirm that Alibaba Cloud created the PodVM:

You can verify that the peer-pod VM was created by running the following command:

aliyun ecs DescribeInstances --RegionId ${REGION_ID} --InstanceName 'podvm-*'

The output should include the VM associated with the nginx pod. If you run into problems, see the troubleshooting guide.

Uninstall

First remove the workloads and uninstall CAA.

Verify that all peer-pod VMs are deleted:

aliyun ecs DescribeInstances --RegionId ${REGION_ID} --InstanceName 'podvm-*'

Delete the ACK cluster by running the following command:

aliyun cs DELETE /clusters/${CLUSTER_ID} --region ${REGION_ID} --keep_slb false --retain_all_resources false --header "Content-Type=application/json;" --body "{}"

2.2 - AWS (EKS)

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on AWS Elastic Kubernetes Service (EKS). It explains how to deploy:

• A single worker node Kubernetes cluster using Elastic Kubernetes Service (EKS)
• CAA on that Kubernetes cluster
• A Nginx pod backed by CAA pod VM

Before you begin

Complete the common prerequisites, including the AWS-specific tools. Then return here to prepare AWS.

EKS Preparation

• Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or AWS_PROFILE) and AWS_REGION for AWS CLI access

Note: As an alternative to static credentials, you can use IRSA (IAM Roles for Service Accounts) on EKS. With IRSA, the CAA pods authenticate via OIDC — no static AWS keys are stored in Kubernetes secrets. You still need AWS_REGION and temporary credentials for the cluster setup steps below.

• Set the region:

• AMD SEV-SNP
• Non-Confidential

export AWS_REGION="us-east-2"

export AWS_REGION="us-east-2"

Setup cluster

Make changes to the following environment variable as you see fit:

export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
export CLUSTER_NODE_TYPE="m5.xlarge"
export CLUSTER_NODE_FAMILY_TYPE="Ubuntu2204"
export SSH_KEY=~/.ssh/id_rsa.pub

Example EKS cluster creation using the default AWS VPC-CNI

eksctl create cluster --name "$CLUSTER_NAME" \
    --node-type "$CLUSTER_NODE_TYPE" \
    --node-ami-family "$CLUSTER_NODE_FAMILY_TYPE" \
    --nodes 1 \
    --nodes-min 0 \
    --nodes-max 2 \
    --node-private-networking \
    --kubeconfig "$CLUSTER_NAME"-kubeconfig

Wait for the cluster to be created.

Use the kubeconfig file created by eksctl for the remaining commands:

export KUBECONFIG="$(pwd)/${CLUSTER_NAME}-kubeconfig"

Label the cluster nodes with node.kubernetes.io/worker=

for NODE_NAME in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label node $NODE_NAME node.kubernetes.io/worker=
done

Allow required network ports

EKS_VPC_ID=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
--query "cluster.resourcesVpcConfig.vpcId" \
--output text)
echo $EKS_VPC_ID

EKS_CLUSTER_SG=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
  --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" \
  --output text)
echo $EKS_CLUSTER_SG

EKS_VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids "$EKS_VPC_ID" \
--query 'Vpcs[0].CidrBlock' --output text)
echo $EKS_VPC_CIDR

# agent-protocol-forwarder port
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 15150 --cidr "$EKS_VPC_CIDR"

# vxlan port
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 9000 --cidr "$EKS_VPC_CIDR"
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol udp --port 9000 --cidr "$EKS_VPC_CIDR"

Note:

• Port 15150 is the default port for CAA to connect to the agent-protocol-forwarder running inside the pod VM.
• Port 9000 is the VXLAN port used by CAA. Ensure it doesn’t conflict with the VXLAN port used by the Kubernetes CNI.

Configure Authentication

Choose how CAA authenticates with AWS. Static credentials use a Kubernetes secret with access keys. IRSA uses OIDC federation so the pods assume an IAM role directly — no static keys needed.

• Static Credentials
• IRSA (EKS)

OIDC_ID=$(aws eks describe-cluster \
  --name ${CLUSTER_NAME} \
  --region ${AWS_REGION} \
  --query "cluster.identity.oidc.issuer" \
  --output text | awk -F'/' '{print $NF}')

aws iam list-open-id-connect-providers | grep ${OIDC_ID}

eksctl utils associate-iam-oidc-provider \
  --cluster ${CLUSTER_NAME} \
  --region ${AWS_REGION} \
  --approve

export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

export OIDC_PROVIDER=$(aws eks describe-cluster \
  --name ${CLUSTER_NAME} \
  --region ${AWS_REGION} \
  --query "cluster.identity.oidc.issuer" \
  --output text | sed 's|https://||')

export NAMESPACE="confidential-containers-system"
export CAA_SERVICE_ACCOUNT="cloud-api-adaptor"
export CAA_ROLE_NAME="CAA-IRSA-Role-${CLUSTER_NAME}"

cat > /tmp/caa-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${NAMESPACE}:${CAA_SERVICE_ACCOUNT}",
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CAA_ROLE_NAME} \
  --assume-role-policy-document file:///tmp/caa-trust-policy.json \
  --description "IRSA role for Cloud API Adaptor on EKS"

aws iam attach-role-policy \
  --role-name ${CAA_ROLE_NAME} \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess

export CAA_ROLE_ARN=$(aws iam get-role \
  --role-name ${CAA_ROLE_NAME} \
  --query 'Role.Arn' \
  --output text)

echo "CAA Role ARN: ${CAA_ROLE_ARN}"

export CTRL_SERVICE_ACCOUNT="peerpodctrl-controller-manager"
export CTRL_ROLE_NAME="PeerpodCtrl-IRSA-Role-${CLUSTER_NAME}"

cat > /tmp/peerpod-ctrl-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${NAMESPACE}:${CTRL_SERVICE_ACCOUNT}",
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

aws iam create-role \
  --role-name ${CTRL_ROLE_NAME} \
  --assume-role-policy-document file:///tmp/peerpod-ctrl-trust-policy.json \
  --description "IRSA role for PeerPod Controller on EKS"

aws iam attach-role-policy \
  --role-name ${CTRL_ROLE_NAME} \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess

export CTRL_ROLE_ARN=$(aws iam get-role \
  --role-name ${CTRL_ROLE_NAME} \
  --query 'Role.Arn' \
  --output text)

echo "Controller Role ARN: ${CTRL_ROLE_ARN}"

Deploy

Preparation

Complete Prepare the CAA deployment.

This downloads the Helm artifacts, selects the CAA image, installs cert-manager, and creates the installation namespace.

Note: Keep the resulting CAA_VERSION, CAA_IMAGE, and CAA_TAG values in your terminal, then return here.

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in AWS.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export PODVM_AMI_ID=$(aws ec2 describe-images \
    --filters Name=name,Values="podvm-ubuntu-amd64-${CAA_VERSION//./-}" \
    --query 'Images[*].[ImageId]' --output text)

echo $PODVM_AMI_ID

Select peer-pods machine type

• AMD SEV-SNP
• Non-Confidential

export PODVM_INSTANCE_TYPE="m6a.large"
export DISABLECVM="false"

export PODVM_INSTANCE_TYPE="t3.large"
export DISABLECVM="true"

Populate the provider file

List of all available configuration options can be found in two places:

• Main charts values
• AWS specific values

Run the following command to update the providers/aws.yaml file:

cat <<EOF > providers/aws.yaml
provider: aws
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   aws:
      DISABLECVM: ${DISABLECVM}
      PODVM_AMI_ID: "${PODVM_AMI_ID}"
      PODVM_INSTANCE_TYPE: "${PODVM_INSTANCE_TYPE}"
      VXLAN_PORT: 9000
EOF

Deploy helm chart

Create credentials and install the Helm chart. The -f and --set customization options are described in advanced configuration.

• Static Credentials
• IRSA (EKS)

kubectl create secret generic my-provider-creds \
  -n confidential-containers-system \
  --from-literal=AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
  --from-literal=AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
  --from-file=id_rsa.pub=${SSH_KEY}

helm install peerpods . \
  -f providers/aws.yaml \
  --set secrets.mode=reference \
  --set secrets.existingSecretName=my-provider-creds \
  --dependency-update \
  -n confidential-containers-system

helm install peerpods . \
  -f providers/aws.yaml \
  --set "daemonset.serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${CAA_ROLE_ARN}" \
  --set "resourceCtrl.serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${CTRL_ROLE_ARN}" \
  --dependency-update \
  -n confidential-containers-system

kubectl get serviceaccount cloud-api-adaptor \
  -n confidential-containers-system \
  -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'

CAA_POD=$(kubectl get pods -n confidential-containers-system \
  -l app=cloud-api-adaptor \
  -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n confidential-containers-system ${CAA_POD} -- env | grep AWS

Generic Peer pods Helm charts deployment instructions are also described here.

Verify deployment

Follow Verify the installation and run a sample workload, then confirm that AWS created the PodVM:

You can verify that the peer pod VM was created by running the following command:

aws ec2 describe-instances --filters "Name=tag:Name,Values=podvm*" \
   --query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value | [0]]' --output table

The output should include the VM associated with the nginx pod.

Note: If you run into problems, see the troubleshooting guide.

Uninstall

First remove the workloads and uninstall CAA.

Verify that all peer-pod VMs are deleted. This command lists VMs whose name starts with podvm and their status:

aws ec2 describe-instances --filters "Name=tag:Name,Values=podvm*" \
--query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value | [0], State.Name]' --output table

Delete the EKS cluster by running the following command:

eksctl delete cluster --name=$CLUSTER_NAME 

2.3 - Azure Kubernetes Service (AKS)

This documentation will walk you through setting up Cloud API Adaptor (CAA) (a.k.a. Peer Pods) on Azure Kubernetes Service (AKS).

It explains how to deploy:

• A single worker node Kubernetes cluster using Azure Kubernetes Service (AKS),
• CAA on that Kubernetes cluster,
• A sample application deployed using CAA to verify that everything is working as expected.

Confidential Containers also supports using Azure Key Vault as a resource backend for Trustee. More info

Before you begin

Complete the common prerequisites, including the Azure-specific tools. Then return here to prepare Azure.

AKS Preparation

Azure login

1. Authenticate with Azure by running the following command and following the instructions in the command line:

   az login
   

   Note: If you have access to multiple Azure accounts, make sure to select the one you want to use for this deployment in the command line after running az login.

2. Export the Azure subscription ID into AZ_SUBSCR_ID using the subscription currently selected in the Azure CLI context:

   export AZ_SUBSCR_ID=$(az account show --query id --output tsv)
   

   Verify that the correct subscription is selected

   az account show --query '{name:name, id:id, tenantId:tenantId}' -o yaml
   

3. Set the AZURE_REGION environment variable to the region you want to use for this deployment. This region will be used to deploy both the AKS cluster and the peer pod VMs.

   • AMD SEV-SNP
   • Intel® TDX
   • Non-Confidential

   export AZURE_REGION="eastus"
   

   Note: We selected the eastus region as it not only offers AMD SEV-SNP machines but also has prebuilt pod VM images readily available.

   export AZURE_REGION="eastus"
   

   Note: We selected the eastus region as it not only offers Intel® TDX machines but also has prebuilt pod VM images readily available.

   Note: Currently Intel® TDX machines are available in a limited number of regions:
   (westeurope, westus, WestUS3, eastus, EastUS2EUAP, northcentralus), so make sure to select a region that supports them.

   export AZURE_REGION="eastus"
   

   Note: We have chose region eastus because it has prebuilt pod VM images readily available.

Resource group

Note: Skip this step if you already have a resource group you want to use. Please, export the resource group name in the AZURE_RESOURCE_GROUP environment variable.

Create an Azure resource group by running the following command:

1. Export the AZURE_RESOURCE_GROUP environment variable to a unique name for the resource group:

   export AZURE_RESOURCE_GROUP="caa-rg-$(date '+%Y%m%b%d%H%M%S')"
   

2. Create the resource group in the specified region:

   az group create \
     --name "${AZURE_RESOURCE_GROUP}" \
     --location "${AZURE_REGION}"
   

Setup cluster

1. Export environment variables related to the AKS cluster deployment:

   export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
   export AKS_WORKER_USER_NAME="azuser"
   export AKS_RG="${AZURE_RESOURCE_GROUP}-aks"
   
   # SSH key is optional, but it allows user to SSH into the pod VMs for troubleshooting purposes. 
   # This option works only for custom debug enabled pod VM images. 
   # The prebuilt pod VM images do not have SSH connection enabled.
   export SSH_KEY=~/.ssh/id_rsa.pub
   

2. Deploy AKS with single worker node to the same resource group you have created:

   az aks create \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --node-resource-group "${AKS_RG}" \
     --name "${CLUSTER_NAME}" \
     --enable-oidc-issuer \
     --enable-workload-identity \
     --location "${AZURE_REGION}" \
     --node-count 1 \
     --node-vm-size Standard_F4s_v2 \
     --nodepool-labels node.kubernetes.io/worker= \
     --ssh-access disabled \
     --admin-username "${AKS_WORKER_USER_NAME}" \
     --os-sku Ubuntu
   

   Note: Optionally, deploy the worker nodes into an existing Azure Virtual Network (VNet) and subnet by adding the following flag: --vnet-subnet-id <MY_SUBNET_ID>.

   Note: For sample usage Standard_F4s_v2 machine type is used for the worker node - this machine is General purpose - without enabled confidentiality.

3. Download kubeconfig locally to access the cluster using kubectl:

   az aks get-credentials \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --name "${CLUSTER_NAME}"
   

User assigned identity and federated credentials

CAA needs privileges to talk to Azure API. This privilege is granted to CAA by associating a workload identity to the CAA service account. This workload identity (a.k.a. user assigned identity) is given permissions to create VMs, fetch images and join networks in the next step.

Note: If you use an existing AKS cluster it might need to be configured to support workload identity and OpenID Connect (OIDC), please refer to the instructions in this guide.

Create an identity for CAA:

1. Set the AZURE_WORKLOAD_IDENTITY_NAME environment variable to a name for the user assigned identity:

   export AZURE_WORKLOAD_IDENTITY_NAME="${CLUSTER_NAME}-identity"
   

2. Create the user assigned identity in Azure:

   az identity create \
     --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --location "${AZURE_REGION}"
   

3. Export the client ID of the user assigned identity to the environment variable USER_ASSIGNED_CLIENT_ID:

   export USER_ASSIGNED_CLIENT_ID="$(az identity show \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --query 'clientId' \
     -o tsv)"
   

Networking

The VMs that will host Pods will commonly require access to internet services, e.g. to pull images from a public OCI registry. Create a discrete subnet next to the AKS cluster subnet in the same VNet for peer pod VMs. Then create a NAT gateway, attach a public IP, and associate the NAT gateway with the peer-pods subnet to provide outbound connectivity.

1. Get the VNet name of the AKS cluster:

   export AZURE_VNET_NAME="$(az network vnet list \
                               -g ${AKS_RG} \
                               --query '[].name' \
                               -o tsv)"
   

2. Define the AKS subnet name.

   By default, the instructions assume aks-subnet, but this can differ depending on how the cluster was created.

   export AKS_SUBNET_NAME="aks-subnet"
   

   Show how to find subnet names in the VNet

   az network vnet subnet list \
     -g "${AKS_RG}" \
     --vnet-name "${AZURE_VNET_NAME}" \
     --query '[].name' \
     -o tsv
   

3. Get the CIDR of the AKS cluster subnet:

   export AKS_CIDR="$(az network vnet show \
     -n $AZURE_VNET_NAME \
     -g $AKS_RG \
     --query "subnets[?name == '${AKS_SUBNET_NAME}'].addressPrefix" \
     -o tsv)"
   

   Expected output example: 10.224.0.0/16.

4. Calculate the CIDR mask for the peer pod subnet by extracting the mask from the AKS CIDR:

   export MASK="${AKS_CIDR#*/}"
   

   Expected output example: 16.

5. Create a CIDR for the peer pod subnet by using sipcalc to calculate a new subnet from the AKS CIDR:

   PEERPOD_CIDR="$(
     sipcalc $AKS_CIDR -n 2 \
     | grep ^Network \
     | grep -v current \
     | cut -d' ' -f2
   )/${MASK}"
   

   Expected output example: 10.225.0.0/16.

6. Create a public IP for the NAT gateway:

   az network public-ip create -g "$AKS_RG" -n peerpod
   

7. Create a NAT gateway and attach the public IP to it:

   az network nat gateway create \
     -g "$AKS_RG" \
     -l "$AZURE_REGION" \
     --public-ip-addresses peerpod \
     -n peerpod
   

8. Create a subnet for the peer pods and attach the NAT gateway to it:

   az network vnet subnet create -g "$AKS_RG" \
     --vnet-name "$AZURE_VNET_NAME" \
     --nat-gateway peerpod \
     --address-prefixes "$PEERPOD_CIDR" \
     -n peerpod
   

9. Export the subnet ID to the environment variable AZURE_SUBNET_ID:

   export AZURE_SUBNET_ID="$(
     az network vnet subnet show \
     -g "$AKS_RG" \
     --vnet-name "$AZURE_VNET_NAME" \
     -n peerpod \
     --query id \
     -o tsv)"
   

AKS resource group permissions

For CAA to be able to manage VMs, assign Virtual Machine and Network related roles to the user assigned identity. The roles grant permissions to create VMs in AZURE_RESOURCE_GROUP and attach them to the VNet/subnet in AKS_RG.

1. Assign the “Virtual Machine Contributor” role to the user assigned identity:

   az role assignment create \
     --role "Virtual Machine Contributor" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"
   

2. Assign the “Reader” role to the user assigned identity:

   az role assignment create \
     --role "Reader" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"
   

3. Assign the “Network Contributor” role to the user assigned identity:

   az role assignment create \
     --role "Network Contributor" \
     --assignee "$USER_ASSIGNED_CLIENT_ID" \
     --scope "/subscriptions/${AZ_SUBSCR_ID}/resourcegroups/${AKS_RG}"
   

Create the federated credential for the CAA ServiceAccount using the OIDC endpoint from the AKS cluster:

1. Set the AKS_OIDC_ISSUER environment variable to the OIDC issuer URL of your AKS cluster:

   export AKS_OIDC_ISSUER="$(az aks show \
     --name "${CLUSTER_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --query "oidcIssuerProfile.issuerUrl" \
     -o tsv)"
   

2. Create the federated credential:

   az identity federated-credential create \
     --name "${CLUSTER_NAME}-federated" \
     --identity-name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
     --resource-group "${AZURE_RESOURCE_GROUP}" \
     --issuer "${AKS_OIDC_ISSUER}" \
     --subject system:serviceaccount:confidential-containers-system:cloud-api-adaptor \
     --audience api://AzureADTokenExchange
   

Deploy

Preparation

Note: If you are using Calico Container Network Interface (CNI) on the Kubernetes cluster, then, configure Virtual Extensible LAN (VXLAN) encapsulation for all inter workload traffic.

Complete Prepare the CAA deployment. This downloads the Helm artifacts, selects the CAA image, installs cert-manager, and creates the installation namespace.

Note: Keep the resulting CAA_VERSION, CAA_IMAGE, and CAA_TAG values in your terminal, then return here.

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Azure.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export AZURE_IMAGE_ID="/CommunityGalleries/cococommunity-42d8482d-92cd-415b-b332-7648bd978eff/Images/peerpod-podvm-fedora/Versions/${CAA_VERSION}"

SUCCESS_TIME=$(curl -s \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/confidential-containers/cloud-api-adaptor/actions/workflows/azure-nightly-build.yml/runs?status=success" \
  | jq -r '.workflow_runs[0].updated_at')

export AZURE_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/$(date -u -jf "%Y-%m-%dT%H:%M:%SZ" "$SUCCESS_TIME" "+%Y.%m.%d" 2>/dev/null || date -d "$SUCCESS_TIME" +%Y.%m.%d)"

Select peer-pods machine type

• AMD SEV-SNP
• Intel® TDX
• Non-Confidential

export AZURE_INSTANCE_SIZE="Standard_DC2as_v5"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_DC2es_v6"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_D2as_v5"
export DISABLECVM="true"

Populate the provider file

List of all available configuration options can be found in two places:

• Main charts values
• Azure specific values

Run the following command to update the providers/azure.yaml file:

cat <<EOF > providers/azure.yaml
provider: azure
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   azure:
      AZURE_IMAGE_ID: "${AZURE_IMAGE_ID}"
      AZURE_REGION: "${AZURE_REGION}"
      AZURE_RESOURCE_GROUP: "${AZURE_RESOURCE_GROUP}"
      AZURE_SUBNET_ID: "${AZURE_SUBNET_ID}"
      AZ_SUBSCR_ID: "${AZ_SUBSCR_ID}"
      AZURE_INSTANCE_SIZE: "${AZURE_INSTANCE_SIZE}"
      DISABLECVM: ${DISABLECVM}
EOF

Deploy helm chart

1. Create a Kubernetes Secret that stores the Azure service-account credentials:

   See providers/azure-secrets.yaml.template for required keys.

   Note: Below example assumes that you are using workload identity for authentication hence AZURE_CLIENT_SECRET and AZURE_TENANT_ID are not provided.

   kubectl create secret generic my-provider-creds \
     -n confidential-containers-system \
     --from-literal=AZURE_CLIENT_ID="${USER_ASSIGNED_CLIENT_ID}" \
     --from-file=id_rsa.pub=${SSH_KEY}
   

   Note: --from-file=id_rsa.pub=${SSH_KEY} is optional. It allows user to SSH into the pod VMs for troubleshooting purposes. This option works only for custom debug enabled pod VM images. The prebuilt pod VM images do not have SSH connection enabled.

2. Install the Helm chart:

   The -f and --set customization options are described in advanced configuration.

   helm install peerpods . \
     -f providers/azure.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' \
     --dependency-update \
     -n confidential-containers-system
   

   Note: Above example assumes that you are using workload identity for authentication.
   This line: --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' is required only when using workload identity.

Generic Peer pods Helm charts deployment instructions are also described here.

Verify deployment

Follow Verify the installation and run a sample workload, then confirm that Azure created the PodVM:

az vm list \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --output table

The output should include the VM associated with the nginx pod.

Note: If you run into problems, see the troubleshooting guide.

Pod VM reference values

Reference values for the PCRs can be retrieved from the registry and used in attestation policies to verify the integrity of the Pod VM image before trusting it with sensitive workloads or secrets.

As part of a Pod VM image build expected PCR measurements are published into an OCI registry.

Pre-requisites

Install ORAS tool to pull the reference values from the OCI registry and GitHub CLI to verify its build provenance.

Verify build provenance

Assert that the measurements have been generated by a trusted build process on the official repository. Specify --format=json to get more details about the build process.

CAA_REPO="confidential-containers/cloud-api-adaptor"
OCI_REGISTRY="ghcr.io/${CAA_REPO}/measurements/azure/podvm:${CAA_VERSION}"
gh attestation verify -R "$CAA_REPO" "oci://${OCI_REGISTRY}"

Retrieve reference values

The PCR values can be used in remote attestation policies to assert the integrity of the PodVM image.

$ oras pull "$OCI_REGISTRY"
$ jq -r .measurements.sha256.pcr11 < measurements.json
0x58e8afdf5b105fc6b202eb8e537a9f1512a4b33cd5921171b518645a86ca5a75

Uninstall

First remove the workloads and uninstall CAA. Then delete the resource group, including the AKS cluster and the Azure resources created by this guide:

az group delete \
  --name "${AZURE_RESOURCE_GROUP}" \
  --yes --no-wait

2.4 - Google Cloud Platform (GCP)

This documentation will walk you through setting up Cloud API Adaptor (CAA) (a.k.a. Peer Pods) on Google Kubernetes Engine (GKE).

It explains how to deploy:

• A single worker node Kubernetes cluster using Google Kubernetes Engine (GKE),
• CAA on that Kubernetes cluster,
• A sample application deployed using CAA to verify that everything is working as expected.

Before you begin

Complete the common prerequisites, including the Google Cloud CLI.

Google Cloud Project:

• Ensure you have a Google Cloud project created,
• Note the Project ID (export it as GCP_PROJECT_ID).

GKE Preparation

1. Set the environment variable GCP_PROJECT_ID to your Google Cloud project ID:

   export GCP_PROJECT_ID="YOUR_PROJECT_ID"
   

2. Authenticate with Google Cloud and set the project:

   gcloud auth login
   gcloud config set project "${GCP_PROJECT_ID}"
   

3. Enable the GKE API:

   gcloud services enable container.googleapis.com \
     --project="${GCP_PROJECT_ID}"
   

   This API is required to create and manage the GKE cluster.

4. Set the region and cluster name:

   export GCP_REGION="us-central1"
   export CLUSTER_NAME="caa-gke"
   

   Note

   “us-central1” was chosen because supports Confidential VMs.
   For a complete list of supported regions visit supported-configurations.

Setup cluster

Deploy a single node Kubernetes cluster using GKE:

gcloud container clusters create "${CLUSTER_NAME}" \
  --zone ${GCP_REGION}-a \
  --machine-type "e2-standard-4" \
  --image-type UBUNTU_CONTAINERD \
  --num-nodes 1

Note: The e2-standard-4 machine type is used for the GKE cluster nodes, which is a general-purpose machine.
The UBUNTU_CONTAINERD image type is specified to ensure compatibility with the container runtime used by CAA.

Get cluster credentials:

gcloud container clusters get-credentials "${CLUSTER_NAME}" \
  --zone "${GCP_REGION}-a" \
  --project "${GCP_PROJECT_ID}"

(Optional) Verify that the cluster is reachable:

kubectl get nodes -o wide

Label the worker node:

kubectl get nodes \
  --selector='!node-role.kubernetes.io/master' \
  -o name \
  | xargs -I{} kubectl label {} node.kubernetes.io/worker=

This labeling step adds the node.kubernetes.io/worker label to non-control-plane nodes.

Configure VPC network

We need to make sure port 15150 is open under the default VPC network:

gcloud compute firewall-rules create allow-port-15150 \
    --project=${GCP_PROJECT_ID} \
    --network=default \
    --allow=tcp:15150

For production scenarios, it is advisable to restrict the source IP range to minimize security risks. For example, you can restrict the source range to a specific IP address or CIDR block:

gcloud compute firewall-rules create allow-port-15150-restricted \
   --project=${GCP_PROJECT_ID} \
   --network=default \
   --allow=tcp:15150 \
   --source-ranges=[YOUR_EXTERNAL_IP]

Create the PodVM image

This section describes how to create the PodVM image for your workload and publish it on Google Cloud Storage.

Build the PodVM image

• Build methods:
• Release
• Debug
• Pre-built

Publish image to Google Storage

1. Export the bucket name:

   export BUCKET_NAME="peerpods-bucket"
   

   Note: GCP_PROJECT_ID and GCP_REGION should already be set from the GCP Preparation section.
   The BUCKET_NAME should be globally unique across all of Google Cloud, so consider adding a random suffix if needed.

2. Create a GCS bucket for images:

   gcloud storage buckets create "gs://${BUCKET_NAME}" \
     --project="${GCP_PROJECT_ID}" \
     --location="${GCP_REGION}"
   

3. Upload the disk image to a bucket and create the image:

   1. Prepare image name:

      export IMAGE_BASE_NAME="podvm-image"
      export CAA_HASH=$(git rev-parse --short HEAD 2>/dev/null || echo "")
      export IMAGE_NAME="${IMAGE_BASE_NAME}-${CAA_HASH:-$(date +%s)}-release"
      

      Note: The image name uses the git commit hash if available (for Release/Debug builds), or falls back to a Unix timestamp (for Pre-built images).

   2. Upload image to GCS bucket:

      gcloud storage cp build/disk.tar.gz gs://${BUCKET_NAME}/peerpods-disk.tar.gz
      

   3. Create image in GCP with defined name from the uploaded disk image:

      • AMD SEV-SNP
      • Intel® TDX

      gcloud compute images create ${IMAGE_NAME} \
        --source-uri=gs://${BUCKET_NAME}/peerpods-disk.tar.gz \
        --guest-os-features=UEFI_COMPATIBLE
      

      This command creates a new image in GCP with the specified name and the uploaded disk image.
      The --guest-os-features flag ensures that the image is compatible with UEFI.

      gcloud compute images create ${IMAGE_NAME} \
        --source-uri=gs://${BUCKET_NAME}/peerpods-disk.tar.gz \
        --guest-os-features=UEFI_COMPATIBLE,TDX_CAPABLE
      

      Both UEFI_COMPATIBLE and TDX_CAPABLE are required for tdx TEE.

      This command creates a new image in GCP with the specified name and the uploaded disk image.
      The --guest-os-features flag ensures that the image is compatible with UEFI and TDX.

Export PodVM image id

Export the PodVM image id to be used in the provider configuration. This is the name of the image created in the previous step.

export PODVM_IMAGE_ID="$IMAGE_NAME"

gcloud compute images list \
  --project="${GCP_PROJECT_ID}" \
  --filter="name ~ ^${IMAGE_BASE_NAME}-" \
  --sort-by=~creationTimestamp \
  --limit=1 \
  --format="value(name)"

Deploy

Preparation

Complete Prepare the CAA deployment.

This downloads the Helm artifacts, selects the CAA image, installs cert-manager, and creates the installation namespace.

Note: Keep the resulting CAA_VERSION, CAA_IMAGE, and CAA_TAG values in your terminal, then return here.

Set TEE Platform Configuration

Set TEE platform and PodVM instance type for your workload:

• AMD SEV-SNP
• Intel® TDX
• Non-Confidential

export PODVM_INSTANCE_TYPE="n2d-standard-4"
export DISABLECVM=false
export GCP_CONFIDENTIAL_TYPE="SEV" # SEV or SEV_SNP
export GCP_DISK_TYPE="pd-standard"

export PODVM_INSTANCE_TYPE="c3-standard-4"
export DISABLECVM=false
export GCP_CONFIDENTIAL_TYPE="TDX"
export GCP_DISK_TYPE="pd-balanced"

export PODVM_INSTANCE_TYPE="e2-medium"
export DISABLECVM=true
export GCP_CONFIDENTIAL_TYPE=""
export GCP_DISK_TYPE="pd-standard"

Populate the provider file

List of all available configuration options can be found in two places:

• Main charts values
• GCP specific values

Run the following command to update the providers/gcp.yaml file:

cat <<EOF > providers/gcp.yaml
provider: gcp
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
  gcp:
    GCP_NETWORK: "global/networks/default"
    GCP_PROJECT_ID: "${GCP_PROJECT_ID}"
    GCP_ZONE: "${GCP_REGION}-a"
    GCP_MACHINE_TYPE: "${PODVM_INSTANCE_TYPE}"
    GCP_DISK_TYPE: "${GCP_DISK_TYPE}"
    PODVM_IMAGE_NAME: "${PODVM_IMAGE_ID}"
    GCP_CONFIDENTIAL_TYPE: "${GCP_CONFIDENTIAL_TYPE}"
    DISABLECVM: ${DISABLECVM}
EOF

Configure Authentication

Choose how CAA authenticates with GCP. You can use either static credentials (service account key file) or Workload Identity Federation (WIF).

Note: Workload Identity Federation (WIF) is the recommended authentication method for GKE deployments. With WIF, the CAA pods authenticate via OIDC — no static GCP credentials are stored in Kubernetes secrets.

Create GCP Service Account

Create a GCP service account with the necessary permissions for CAA to manage peer pod VMs:

export GSA_NAME="cloud-api-adaptor"
export GSA_EMAIL="${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com"

gcloud iam service-accounts create ${GSA_NAME} \
  --description="Cloud API Adaptor Service Account" \
  --display-name="Cloud API Adaptor Service Account" \
  --project=${GCP_PROJECT_ID}

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/compute.instanceAdmin.v1"

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/iam.serviceAccountUser"

These roles allow the Cloud API Adaptor to:

• create, start/stop, and delete the Compute Engine instances used as PodVMs (roles/compute.instanceAdmin.v1)
• run actions as the service account when provisioning those resources (roles/iam.serviceAccountUser)

Note: IAM policy updates can take a few minutes to propagate. If later steps fail with permission errors, wait briefly and retry.

Choose Authentication Method

• Static Credentials
• Workload Identity Federation (GKE)

export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/peerpods_application_key.json

gcloud iam service-accounts keys create \
  ${GOOGLE_APPLICATION_CREDENTIALS} \
  --iam-account=${GSA_EMAIL}

gcloud services enable compute.googleapis.com iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \
  --project=${GCP_PROJECT_ID}

export CAA_K8S_SERVICE_ACCOUNT="cloud-api-adaptor"
export CAA_NAMESPACE="confidential-containers-system"
export GKE_WORKLOAD_POOL="${GCP_PROJECT_ID}.svc.id.goog"

gcloud container clusters describe ${CLUSTER_NAME} \
  --zone=${GCP_REGION}-a \
  --format="value(workloadIdentityConfig.workloadPool)"

gcloud container clusters update ${CLUSTER_NAME} \
  --zone=${GCP_REGION}-a \
  --workload-pool=${GKE_WORKLOAD_POOL}

# Note: Use 'locations' instead of 'zones' for the OIDC issuer URL
export OIDC_ISSUER="https://container.googleapis.com/v1/projects/${GCP_PROJECT_ID}/locations/${GCP_REGION}-a/clusters/${CLUSTER_NAME}"

export PROJECT_NUMBER=$(gcloud projects describe ${GCP_PROJECT_ID} \
  --format="value(projectNumber)")

echo "OIDC Issuer: ${OIDC_ISSUER}"
echo "Project Number: ${PROJECT_NUMBER}"

# Verify the OIDC endpoint is accessible
curl -s "${OIDC_ISSUER}/.well-known/openid-configuration" | jq .issuer

export WI_POOL_NAME="caa-direct-wif-pool"

gcloud iam workload-identity-pools create ${WI_POOL_NAME} \
  --location=global \
  --description="Workload Identity pool for cloud-api-adaptor with hostNetwork true" \
  --display-name="CAA Direct WIF Pool" \
  --project=${GCP_PROJECT_ID}

export OIDC_PROVIDER_NAME="${CLUSTER_NAME}-oidc-provider"

gcloud iam workload-identity-pools providers create-oidc ${OIDC_PROVIDER_NAME} \
  --project=${GCP_PROJECT_ID} \
  --location=global \
  --workload-identity-pool=${WI_POOL_NAME} \
  --issuer-uri="${OIDC_ISSUER}" \
  --allowed-audiences="${GKE_WORKLOAD_POOL}" \
  --attribute-mapping="google.subject=assertion.sub,attribute.namespace=assertion['kubernetes.io']['namespace'],attribute.service_account_name=assertion['kubernetes.io']['serviceaccount']['name']" \
  --attribute-condition="assertion.sub.startsWith('system:serviceaccount:')"

gcloud iam service-accounts add-iam-policy-binding ${GSA_EMAIL} \
  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${WI_POOL_NAME}/subject/system:serviceaccount:${CAA_NAMESPACE}:${CAA_K8S_SERVICE_ACCOUNT}" \
  --role="roles/iam.workloadIdentityUser" \
  --project=${GCP_PROJECT_ID}

gcloud iam service-accounts add-iam-policy-binding ${GSA_EMAIL} \
  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${WI_POOL_NAME}/subject/system:serviceaccount:${CAA_NAMESPACE}:${CAA_K8S_SERVICE_ACCOUNT}" \
  --role="roles/iam.serviceAccountTokenCreator" \
  --project=${GCP_PROJECT_ID}

export PROVIDER_NAME="//iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${WI_POOL_NAME}/providers/${OIDC_PROVIDER_NAME}"

echo "Provider Name: ${PROVIDER_NAME}"
echo "GSA Email: ${GSA_EMAIL}"

Deploy helm chart

1. Create a ResourceQuota to allow system priority classes:

   GKE enforces strict resource quota policies. The kata-deploy DaemonSet requires system priority classes (system-node-critical or system-cluster-critical) to ensure proper scheduling and execution. Create a ResourceQuota to allow these priority classes:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: allow-system-priority-classes
     namespace: confidential-containers-system
   spec:
     hard:
       pods: "100"
     scopeSelector:
       matchExpressions:
       - operator: In
         scopeName: PriorityClass
         values:
         - system-node-critical
         - system-cluster-critical
   EOF
   

2. Create credentials and install the Helm chart:

   The -f and --set customization options are described in advanced configuration.

• Static Credentials
• Workload Identity Federation (GKE)

kubectl create secret generic my-provider-creds \
  -n confidential-containers-system \
  --from-file=GCP_CREDENTIALS="${GOOGLE_APPLICATION_CREDENTIALS}"

helm install peerpods . \
  -f providers/gcp.yaml \
  --set secrets.mode=reference \
  --set secrets.existingSecretName=my-provider-creds \
  --dependency-update \
  -n confidential-containers-system

cat > providers/gcp-wif-values.yaml <<EOF
# WIF-specific configuration
gcp:
  workloadIdentityFederation:
    enable: true
    serviceAccount: "${GSA_EMAIL}"
    workloadPool: "${GKE_WORKLOAD_POOL}"
    cluster: "${PROVIDER_NAME}"
EOF

helm install peerpods . \
  -f providers/gcp.yaml \
  -f providers/gcp-wif-values.yaml \
  --dependency-update \
  -n confidential-containers-system

kubectl get secret peer-pods-secret \
  -n confidential-containers-system \
  -o jsonpath='{.data.GCP_WIF_CREDENTIALS}' | base64 -d | jq

CAA_POD=$(kubectl get pods -n confidential-containers-system \
  -l app=cloud-api-adaptor \
  -o jsonpath='{.items[0].metadata.name}')

kubectl logs -n confidential-containers-system ${CAA_POD} | grep "GCP_WIF_CREDENTIALS"

Generic Peer pods Helm charts deployment instructions are also described here.

Verify deployment

Follow the shared installation verification. You can then use either sample below.

Run sample application

• CoCo Secret Retrieval
• Basic nginx

# initdata.toml
algorithm = "sha384"
version = "0.1.0"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]
url = 'http://127.0.0.1:8080'

[token_configs.kbs]
url = 'http://127.0.0.1:8080'
cert = """
-----BEGIN CERTIFICATE-----
MIIDljCCAn6gAwIBAgIUR/UNh13GFam4emgludtype/S9BIwDQYJKoZIhvcNAQEL
BQAwdTELMAkGA1UEBhMCQ04xETAPBgNVBAgMCFpoZWppYW5nMREwDwYDVQQHDAhI
YW5nemhvdTERMA8GA1UECgwIQUFTLVRFU1QxFDASBgNVBAsMC0RldmVsb3BtZW50
MRcwFQYDVQQDDA5BQVMtVEVTVC1IVFRQUzAeFw0yNDAzMTgwNzAzNTNaFw0yNTAz
MTgwNzAzNTNaMHUxCzAJBgNVBAYTAkNOMREwDwYDVQQIDAhaaGVqaWFuZzERMA8G
A1UEBwwISGFuZ3pob3UxETAPBgNVBAoMCEFBUy1URVNUMRQwEgYDVQQLDAtEZXZl
bG9wbWVudDEXMBUGA1UEAwwOQUFTLVRFU1QtSFRUUFMwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDfp1aBr6LiNRBlJUcDGcAbcUCPG6UzywtVIc8+comS
ay//gwz2AkDmFVvqwI4bdp/NUCwSC6ShHzxsrCEiagRKtA3af/ckM7hOkb4S6u/5
ewHHFcL6YOUp+NOH5/dSLrFHLjet0dt4LkyNBPe7mKAyCJXfiX3wb25wIBB0Tfa0
p5VoKzwWeDQBx7aX8TKbG6/FZIiOXGZdl24DGARiqE3XifX7DH9iVZ2V2RL9+3WY
05GETNFPKtcrNwTy8St8/HsWVxjAzGFzf75Lbys9Ff3JMDsg9zQzgcJJzYWisxlY
g3CmnbENP0eoHS4WjQlTUyY0mtnOwodo4Vdf8ZOkU4wJAgMBAAGjHjAcMBoGA1Ud
EQQTMBGCCWxvY2FsaG9zdIcEfwAAATANBgkqhkiG9w0BAQsFAAOCAQEAKW32spii
t2JB7C1IvYpJw5mQ5bhIlldE0iB5rwWvNbuDgPrgfTI4xiX5sumdHw+P2+GU9KXF
nWkFRZ9W/26xFrVgGIS/a07aI7xrlp0Oj+1uO91UhCL3HhME/0tPC6z1iaFeZp8Y
T1tLnafqiGiThFUgvg6PKt86enX60vGaTY7sslRlgbDr9sAi/NDSS7U1PviuC6yo
yJi7BDiRSx7KrMGLscQ+AKKo2RF1MLzlJMa1kIZfvKDBXFzRd61K5IjDRQ4HQhwX
DYEbQvoZIkUTc1gBUWDcAUS5ztbJg9LCb9WVtvUTqTP2lGuNymOvdsuXq+sAZh9b
M9QaC1mzQ/OStg==
-----END CERTIFICATE-----
"""
'''

"cdh.toml"  = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = 'http://1.2.3.4:8080'
kbs_cert = """
-----BEGIN CERTIFICATE-----
MIIFTDCCAvugAwIBAgIBADBGBgkqhkiG9w0BAQowOaAPMA0GCWCGSAFlAwQCAgUA
oRwwGgYJKoZIhvcNAQEIMA0GCWCGSAFlAwQCAgUAogMCATCjAwIBATB7MRQwEgYD
VQQLDAtFbmdpbmVlcmluZzELMAkGA1UEBhMCVVMxFDASBgNVBAcMC1NhbnRhIENs
YXJhMQswCQYDVQQIDAJDQTEfMB0GA1UECgwWQWR2YW5jZWQgTWljcm8gRGV2aWNl
czESMBAGA1UEAwwJU0VWLU1pbGFuMB4XDTIzMDEyNDE3NTgyNloXDTMwMDEyNDE3
NTgyNlowejEUMBIGA1UECwwLRW5naW5lZXJpbmcxCzAJBgNVBAYTAlVTMRQwEgYD
VQQHDAtTYW50YSBDbGFyYTELMAkGA1UECAwCQ0ExHzAdBgNVBAoMFkFkdmFuY2Vk
IE1pY3JvIERldmljZXMxETAPBgNVBAMMCFNFVi1WQ0VLMHYwEAYHKoZIzj0CAQYF
K4EEACIDYgAExmG1ZbuoAQK93USRyZQcsyobfbaAEoKEELf/jK39cOVJt1t4s83W
XM3rqIbS7qHUHQw/FGyOvdaEUs5+wwxpCWfDnmJMAQ+ctgZqgDEKh1NqlOuuKcKq
2YAWE5cTH7sHo4IBFjCCARIwEAYJKwYBBAGceAEBBAMCAQAwFwYJKwYBBAGceAEC
BAoWCE1pbGFuLUIwMBEGCisGAQQBnHgBAwEEAwIBAzARBgorBgEEAZx4AQMCBAMC
AQAwEQYKKwYBBAGceAEDBAQDAgEAMBEGCisGAQQBnHgBAwUEAwIBADARBgorBgEE
AZx4AQMGBAMCAQAwEQYKKwYBBAGceAEDBwQDAgEAMBEGCisGAQQBnHgBAwMEAwIB
CDARBgorBgEEAZx4AQMIBAMCAXMwTQYJKwYBBAGceAEEBEDDhCejDUx6+dlvehW5
cmmCWmTLdqI1L/1dGBFdia1HP46MC82aXZKGYSutSq37RCYgWjueT+qCMBE1oXDk
d1JOMEYGCSqGSIb3DQEBCjA5oA8wDQYJYIZIAWUDBAICBQChHDAaBgkqhkiG9w0B
AQgwDQYJYIZIAWUDBAICBQCiAwIBMKMDAgEBA4ICAQACgCai9x8DAWzX/2IelNWm
ituEBSiq9C9eDnBEckQYikAhPasfagnoWFAtKu/ZWTKHi+BMbhKwswBS8W0G1ywi
cUWGlzigI4tdxxf1YBJyCoTSNssSbKmIh5jemBfrvIBo1yEd+e56ZJMdhN8e+xWU
bvovUC2/7Dl76fzAaACLSorZUv5XPJwKXwEOHo7FIcREjoZn+fKjJTnmdXce0LD6
9RHr+r+ceyE79gmK31bI9DYiJoL4LeGdXZ3gMOVDR1OnDos5lOBcV+quJ6JujpgH
d9g3Sa7Du7pusD9Fdap98ocZslRfFjFi//2YdVM4MKbq6IwpYNB+2PCEKNC7SfbO
NgZYJuPZnM/wViES/cP7MZNJ1KUKBI9yh6TmlSsZZOclGJvrOsBZimTXpATjdNMt
cluKwqAUUzYQmU7bf2TMdOXyA9iH5wIpj1kWGE1VuFADTKILkTc6LzLzOWCofLxf
onhTtSDtzIv/uel547GZqq+rVRvmIieEuEvDETwuookfV6qu3D/9KuSr9xiznmEg
xynud/f525jppJMcD/ofbQxUZuGKvb3f3zy+aLxqidoX7gca2Xd9jyUy5Y/83+ZN
bz4PZx81UJzXVI9ABEh8/xilATh1ZxOePTBJjN7lgr0lXtKYjV/43yyxgUYrXNZS
oLSG2dLCK9mjjraPjau34Q==
-----END CERTIFICATE-----
"""
'''

"policy.rego" = '''
package agent_policy

import future.keywords.in
import future.keywords.every

import input

# Default values, returned by OPA when rules cannot be evaluated to true.
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := false
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default OnlineCPUMemRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StatsContainerRequest := true
default TtyWinResizeRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := false
'''

INITDATA=$(cat initdata.toml | gzip | base64 -w0)

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: "$INITDATA"
spec:
  runtimeClassName: kata-remote
  containers:
    - name: example-container
      image: alpine:latest
      command:
        - sleep
        - "3600"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
EOF

kubectl exec -it example-pod -- curl http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the PodVM was created by running the following command:

gcloud compute instances list

Here you should see the VM associated with the pod used by the example above.

Uninstall

First remove all kata-remote workloads, then verify that all peer-pod VMs are deleted:

Use the following command to list all the peer pod VMs (VMs having prefix podvm) and status.

gcloud compute instances list \
  --filter="name~'podvm.*'" \
  --format="table(name,zone,status)"

After the PodVMs are gone, uninstall CAA.

Remove the GCP service account and its project-level IAM bindings:

gcloud projects remove-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/compute.instanceAdmin.v1"

gcloud projects remove-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/iam.serviceAccountUser"

gcloud iam service-accounts delete ${GSA_EMAIL} \
  --project=${GCP_PROJECT_ID}

Workload Identity Federation only: Delete the OIDC provider and workload identity pool created during setup:

gcloud iam workload-identity-pools providers delete ${OIDC_PROVIDER_NAME} \
  --location=global \
  --workload-identity-pool=${WI_POOL_NAME} \
  --project=${GCP_PROJECT_ID}

gcloud iam workload-identity-pools delete ${WI_POOL_NAME} \
  --location=global \
  --project=${GCP_PROJECT_ID}

Note: Deleting a pool or provider is a soft delete; GCP retains the resource for 30 days before permanently removing it. During this window the name cannot be reused unless you undelete it.

Delete the GKE cluster and confirm deletion when prompted:

gcloud container clusters delete "${CLUSTER_NAME}" \
  --zone "${GCP_REGION}-a"

Debug SSH connection

Note: SSH connection is available only for debug image, which is built with enabled SSH server and added public key to authorized_keys file. If you want to have SSH access to the image, make sure to build debug image and upload it to Google using above instructions.

After creating debug image with enabled SSH, deploy CoCo with sample pod and use root account to access it:

Note: Remember to add your public key to Google using CLI

1. Export environment variable GCP_PODVM_IP using below code:

   GCP_LATEST_PODVM=$(gcloud compute instances list \
                               --project="${GCP_PROJECT_ID}" \
                               --filter="name ~ ^podvm-" \
                               --sort-by=~creationTimestamp \
                               --limit=1 \
                               --format="value(name)")
   
   export GCP_PODVM_IP=$(gcloud compute instances describe "${GCP_LATEST_PODVM}" \
                           --project="${GCP_PROJECT_ID}" \
                           --zone="$(gcloud compute instances list \
                                       --project="${GCP_PROJECT_ID}" \
                                       --filter="name=${GCP_LATEST_PODVM}" \
                                       --format="value(zone)")" \
                           --format="value(networkInterfaces[0].accessConfigs[0].natIP)")
   

2. Connect to debug image using SSH:

   ssh -i ./gcp_ssh_debug root@"$GCP_PODVM_IP"
   

Troubleshooting

Note: If your case is not covered below, see the troubleshooting guide.

VM Doesn’t Start

Starting with GKE version 1.27, GCP configures containerd with the discard_unpacked_layers=true flag to optimize disk usage by removing compressed image layers after they are unpacked. However, this can cause issues with PeerPods, as the workload may fail to locate required layers. To avoid this, disable the discard_unpacked_layers setting in the containerd configuration.

Most of the time you will see a generic message such as the following:

Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:<SHA>: failed to get reader from content store: content digest sha256:<SHA>: not found

To disable the discard_unpacked_layers setting in the containerd configuration on Google Kubernetes Engine (GKE) version 1.27 or later, follow these steps:

1. SSH to worker node Google console

2. Run command which will change the discard_unpacked_layers property to false in the containerd configuration file:

   sudo sed -i 's/discard_unpacked_layers = true/discard_unpacked_layers = false/' /etc/containerd/config.toml
   

3. Verify changed property:

   sudo cat /etc/containerd/config.toml | grep discard_unpacked_layers
   

4. Restart containerd using below command:

   sudo systemctl restart containerd
   

3 - Verify and clean up a CAA deployment

Use these steps after installing the Cloud API Adaptor (CAA) with any cloud-provider guide.

Verify the installation

Confirm that the kata-remote runtime class exists:

kubectl get runtimeclass

A successful deployment returns output similar to:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Run a sample workload

Create a nginx deployment that uses the kata-remote runtime class:

kubectl apply -f - <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      runtimeClassName: kata-remote
      containers:
        - name: nginx
          image: nginx
          ports:
            - containerPort: 80
EOF

Wait for the pod to become ready:

kubectl wait --for=condition=Ready pod -l app=nginx --timeout=10m
kubectl get pods -l app=nginx -o wide

Return to your provider guide for the command that confirms creation of the corresponding PodVM in the cloud console or API.

If the pod does not start, see the troubleshooting guide.

Remove the sample workload

Delete the sample deployment and wait for its PodVM to be removed:

kubectl delete deployment nginx

Before uninstalling CAA, find any other pods that use kata-remote and delete them.

For example, the following command finds all pods that use kata-remote and deletes them:

kubectl get pods -A -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,RUNTIMECLASS:.spec.runtimeClassName' \
    | grep kata-remote \
    | awk '{print $1, $2}' \
    | xargs -n 2 sh -c 'kubectl delete pod -n "$2" "$1"' _

Uninstall CAA

List deployed Confidential Containers Helm chart:

Note: This command assumes that only one Helm release is deployed in the confidential-containers-system namespace. If there are multiple releases, you may need to adjust the command to select the correct one.

export HELM_COCO_CHART_NAME=$(helm list \
                              -n confidential-containers-system \
                              --short)

Delete Confidential Containers related Helm chart:

helm uninstall ${HELM_COCO_CHART_NAME} \
  --namespace confidential-containers-system

Delete Confidential Containers related namespace:

kubectl delete namespace confidential-containers-system

This does not delete the Kubernetes cluster, cloud identities, networks, images, or other provider resources. Return to your provider guide to remove those resources.