AWS

Peer Pods Helm Chart using Cloud API Adaptor (CAA) on AWS

Pre-requisites

Install Required Tools:

Install kubectl,
Install Helm,
Install aws CLI tool,
Install eksctl CLI tool,
Ensure that the tools curl, git and jq are installed.

AWS Preparation

Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or AWS_PROFILE) and AWS_REGION for AWS CLI access

Note: As an alternative to static credentials, you can use IRSA (IAM Roles for Service Accounts) on EKS. With IRSA, the CAA pods authenticate via OIDC — no static AWS keys are stored in Kubernetes secrets. You still need AWS_REGION and temporary credentials for the cluster setup steps below.

Set the region:

export AWS_REGION="us-east-2"

Note: We have chose region us-east-2 as it has AMD SEV-SNP instances as well as prebuilt pod VM images readily available.

export AWS_REGION="us-east-2"

Note: We have chose region us-east-2 because it has prebuilt pod VM images readily available.

Deploy Kubernetes using EKS

Make changes to the following environment variable as you see fit:

export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
export CLUSTER_NODE_TYPE="m5.xlarge"
export CLUSTER_NODE_FAMILY_TYPE="Ubuntu2204"
export SSH_KEY=~/.ssh/id_rsa.pub

Example EKS cluster creation using the default AWS VPC-CNI

eksctl create cluster --name "$CLUSTER_NAME" \
    --node-type "$CLUSTER_NODE_TYPE" \
    --node-ami-family "$CLUSTER_NODE_FAMILY_TYPE" \
    --nodes 1 \
    --nodes-min 0 \
    --nodes-max 2 \
    --node-private-networking \
    --kubeconfig "$CLUSTER_NAME"-kubeconfig

Wait for the cluster to be created.

Label the cluster nodes with node.kubernetes.io/worker=

for NODE_NAME in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label node $NODE_NAME node.kubernetes.io/worker=
done

Allow required network ports

EKS_VPC_ID=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
--query "cluster.resourcesVpcConfig.vpcId" \
--output text)
echo $EKS_VPC_ID

EKS_CLUSTER_SG=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
  --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" \
  --output text)
echo $EKS_CLUSTER_SG

EKS_VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids "$EKS_VPC_ID" \
--query 'Vpcs[0].CidrBlock' --output text)
echo $EKS_VPC_CIDR

# agent-protocol-forwarder port
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 15150 --cidr "$EKS_VPC_CIDR"

# vxlan port
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 9000 --cidr "$EKS_VPC_CIDR"
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol udp --port 9000 --cidr "$EKS_VPC_CIDR"

Note:

Port 15150 is the default port for CAA to connect to the agent-protocol-forwarder running inside the pod VM.

Port 9000 is the VXLAN port used by CAA. Ensure it doesn’t conflict with the VXLAN port used by the Kubernetes CNI.

Configure Authentication

Choose how CAA authenticates with AWS. Static credentials use a Kubernetes secret with access keys. IRSA uses OIDC federation so the pods assume an IAM role directly — no static keys needed.

Ensure AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY (or AWS_PROFILE) are set in your environment. These will be stored in a Kubernetes secret and used by the CAA pods.

No additional setup is needed here — the secret is created during the Helm chart deployment step.

IRSA eliminates the need for long-lived AWS access keys stored in Kubernetes secrets, and is the recommended authentication method for EKS.

Enable OIDC Provider

Check if the IAM OIDC provider is already registered:

OIDC_ID=$(aws eks describe-cluster \
  --name ${CLUSTER_NAME} \
  --region ${AWS_REGION} \
  --query "cluster.identity.oidc.issuer" \
  --output text | awk -F'/' '{print $NF}')

aws iam list-open-id-connect-providers | grep ${OIDC_ID}

If the command returns empty, create the OIDC provider:

eksctl utils associate-iam-oidc-provider \
  --cluster ${CLUSTER_NAME} \
  --region ${AWS_REGION} \
  --approve

Export the account ID and OIDC provider for use in the following steps:

export ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)

export OIDC_PROVIDER=$(aws eks describe-cluster \
  --name ${CLUSTER_NAME} \
  --region ${AWS_REGION} \
  --query "cluster.identity.oidc.issuer" \
  --output text | sed 's|https://||')

Create IAM Role for cloud-api-adaptor

export NAMESPACE="confidential-containers-system"
export CAA_SERVICE_ACCOUNT="cloud-api-adaptor"
export CAA_ROLE_NAME="CAA-IRSA-Role"

Create the trust policy:

cat > /tmp/caa-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${NAMESPACE}:${CAA_SERVICE_ACCOUNT}",
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

Create the IAM role and attach the AmazonEC2FullAccess managed policy:

aws iam create-role \
  --role-name ${CAA_ROLE_NAME} \
  --assume-role-policy-document file:///tmp/caa-trust-policy.json \
  --description "IRSA role for Cloud API Adaptor on EKS"

aws iam attach-role-policy \
  --role-name ${CAA_ROLE_NAME} \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess

Note: AmazonEC2FullAccess grants broad EC2 permissions. For production workloads, it is strongly recommended to replace it with a custom least-privilege policy scoped to the specific EC2 actions CAA requires. See the AWS IRSA documentation for more configuration options.

Export the role ARN for later use:

export CAA_ROLE_ARN=$(aws iam get-role \
  --role-name ${CAA_ROLE_NAME} \
  --query 'Role.Arn' \
  --output text)

echo "CAA Role ARN: ${CAA_ROLE_ARN}"

Create IAM Role for Peerpod-ctrl (optional)

This step is only required if you are deploying Peerpod-ctrl.

export CTRL_SERVICE_ACCOUNT="peerpodctrl-controller-manager"
export CTRL_ROLE_NAME="PeerpodCtrl-IRSA-Role"

Create the trust policy:

cat > /tmp/peerpod-ctrl-trust-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::${ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "${OIDC_PROVIDER}:sub": "system:serviceaccount:${NAMESPACE}:${CTRL_SERVICE_ACCOUNT}",
          "${OIDC_PROVIDER}:aud": "sts.amazonaws.com"
        }
      }
    }
  ]
}
EOF

Create the role and attach permissions:

aws iam create-role \
  --role-name ${CTRL_ROLE_NAME} \
  --assume-role-policy-document file:///tmp/peerpod-ctrl-trust-policy.json \
  --description "IRSA role for PeerPod Controller on EKS"

aws iam attach-role-policy \
  --role-name ${CTRL_ROLE_NAME} \
  --policy-arn arn:aws:iam::aws:policy/AmazonEC2FullAccess

Note: AmazonEC2FullAccess grants broad EC2 permissions. For production workloads, it is strongly recommended to replace it with a custom least-privilege policy scoped to the specific EC2 actions CAA requires. See the AWS IRSA documentation for more configuration options.

Export the role ARN:

export CTRL_ROLE_ARN=$(aws iam get-role \
  --role-name ${CTRL_ROLE_NAME} \
  --query 'Role.Arn' \
  --output text)

echo "Controller Role ARN: ${CTRL_ROLE_ARN}"

Deploy the CAA Helm chart

Download the CAA Helm deployment artifacts

export CAA_VERSION="0.22.0"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

This assumes that you already have the code ready to use. On your terminal change directory to the Cloud API Adaptor’s code base.

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in AWS.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

We have a pre-built debug pod VM image available in us-east-2 for PoCs. You can find the AMI id for the release specific image by running the following CLI:

export PODVM_AMI_ID=$(aws ec2 describe-images \
    --filters Name=name,Values="podvm-ubuntu-amd64-${CAA_VERSION//./-}" \
    --query 'Images[*].[ImageId]' --output text)

echo $PODVM_AMI_ID

There are no pre-built pod VM AMI for latest builds. You’ll need to build your own pod VM image and then create the AMI by following the instructions here.

Remember to set TEE_PLATFORM=amd before building the pod VM image for AWS.

Once image build is finished, export image id to the environment variable PODVM_AMI_ID.

You can build your custom pod VM image by following the instructions here.

Remember to set TEE_PLATFORM=amd before building the pod VM image for AWS.

Once image build is finished, export image id to the environment variable PODVM_AMI_ID.

Export CAA container image path

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

Export the following environment variable to use the latest release image of CAA:

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

Export the following environment variable to use the image built by the CAA CI on each merge to main:

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

Find an appropriate tag of pre-built image suitable to your needs here.

export CAA_TAG=""

Caution: You can also use the latest tag but it is not recommended, because of its lack of version control and potential for unpredictable updates, impacting stability and reproducibility in deployments.

If you have made changes to the CAA code and you want to deploy those changes then follow these instructions to build the container image. Once the image is built export the environment variables CAA_IMAGE and CAA_TAG.

Select peer-pods machine type

export PODVM_INSTANCE_TYPE="m6a.large"
export DISABLECVM="false"

Find more AMD SEV-SNP machine types on this AWS documentation.

export PODVM_INSTANCE_TYPE="t3.large"
export DISABLECVM="true"

Populate the `providers/aws.yaml` file

List of all available configuration options can be found in two places:

Run the following command to update the providers/aws.yaml file:

cat <<EOF > providers/aws.yaml
provider: aws
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   aws:
      DISABLECVM: ${DISABLECVM}
      PODVM_AMI_ID: "${PODVM_AMI_ID}"
      PODVM_INSTANCE_TYPE: "${PODVM_INSTANCE_TYPE}"
      VXLAN_PORT: 9000
EOF

Install cert-manager

CAA requires cert-manager to be installed. Unless you already have it installed deploy it via:

helm repo add jetstack https://charts.jetstack.io
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --set crds.enabled=true \
  --wait \
  --timeout 5m

Deploy helm chart on the Kubernetes cluster

Create namespace managed by Helm:

kubectl apply -f - << EOF
apiVersion: v1
kind: Namespace
metadata:
 name: confidential-containers-system
 labels:
   app.kubernetes.io/managed-by: Helm
 annotations:
   meta.helm.sh/release-name: peerpods
   meta.helm.sh/release-namespace: confidential-containers-system
EOF

Create credentials and install the Helm chart:

Below commands use customization options -f and --set which are described here.

Create the secret using kubectl. See providers/aws-secrets.yaml.template for required keys.

kubectl create secret generic my-provider-creds \
  -n confidential-containers-system \
  --from-literal=AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
  --from-literal=AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
  --from-file=id_rsa.pub=${SSH_KEY}

Note: --from-file=id_rsa.pub=${SSH_KEY} is optional. It allows user to SSH into the pod VMs for troubleshooting purposes. This option works only for custom debug enabled pod VM images. The prebuilt pod VM images do not have SSH connection enabled.

Install the Helm chart:

helm install peerpods . \
  -f providers/aws.yaml \
  --set secrets.mode=reference \
  --set secrets.existingSecretName=my-provider-creds \
  --dependency-update \
  -n confidential-containers-system

When using IRSA, no AWS credentials secret is needed. The CAA pods authenticate via the IAM role configured in the Configure Authentication section.

Install the Helm chart with IRSA annotations:

helm install peerpods . \
  -f providers/aws.yaml \
  --set "daemonset.serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${CAA_ROLE_ARN}" \
  --set "resourceCtrl.serviceAccount.annotations.eks\.amazonaws\.com/role-arn=${CTRL_ROLE_ARN}" \
  --dependency-update \
  -n confidential-containers-system

Note: The resourceCtrl.serviceAccount.annotations line is only required if you are deploying Peerpod-ctrl. If not, omit that --set flag and skip the Peerpod-ctrl IAM role step above.

To verify IRSA is working, check the service account annotation and pod environment:

kubectl get serviceaccount cloud-api-adaptor \
  -n confidential-containers-system \
  -o jsonpath='{.metadata.annotations.eks\.amazonaws\.com/role-arn}'

CAA_POD=$(kubectl get pods -n confidential-containers-system \
  -l app=cloud-api-adaptor \
  -o jsonpath='{.items[0].metadata.name}')

kubectl exec -n confidential-containers-system ${CAA_POD} -- env | grep AWS

The output should include AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN variables.

Generic Peer pods Helm charts deployment instructions are also described here.

Run sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying CAA:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

Create an nginx deployment:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always
EOF

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the peer pod VM was created by running the following command:

aws ec2 describe-instances --filters "Name=tag:Name,Values=podvm*" \
   --query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value | [0]]' --output table

Here you should see the VM associated with the pod nginx.

Note: If you run into problems then check the troubleshooting guide here.

Cleanup

Delete all running pods using the runtimeclass kata-remote. You can use the following command for the same:

kubectl get pods -A -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,RUNTIMECLASS:.spec.runtimeClassName' | grep kata-remote | awk '{print $1, $2}'

Verify that all peer-pod VMs are deleted. You can use the following command to list all the peer-pod VMs (VMs having prefix podvm) and status:

aws ec2 describe-instances --filters "Name=tag:Name,Values=podvm*" \
--query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value | [0], State.Name]' --output table

Delete the EKS cluster by running the following command:

eksctl delete cluster --name=$CLUSTER_NAME

Last modified July 26, 2026: docs: add IRSA authentication guide for CAA on EKS (73c56fc)

AWS

Categories:

Tags:

Pre-requisites

AWS Preparation

Deploy Kubernetes using EKS

Allow required network ports

Configure Authentication

Deploy the CAA Helm chart

Download the CAA Helm deployment artifacts

Export PodVM image version

Export CAA container image path

Select peer-pods machine type

Populate the `providers/aws.yaml` file

Install cert-manager

Deploy helm chart on the Kubernetes cluster

Run sample application

Ensure runtimeclass is present

Deploy workload

Cleanup

AWS

Pre-requisites

AWS Preparation

Deploy Kubernetes using EKS

Allow required network ports

Configure Authentication

Deploy the CAA Helm chart

Download the CAA Helm deployment artifacts

Export PodVM image version

Export CAA container image path

Select peer-pods machine type

Populate the providers/aws.yaml file

Install cert-manager

Deploy helm chart on the Kubernetes cluster

Run sample application

Ensure runtimeclass is present

Deploy workload

Cleanup

Populate the `providers/aws.yaml` file