Examples

• 1: Alibaba Cloud
• 2: AWS
• 3: Azure
• 4: Container Launch with SNP Memory Encryption
• 5: GCP
• 6: NVIDIA confidential NIM deployment
• 7: NVIDIA GPU examples

1 - Alibaba Cloud

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on Alibaba Cloud Container Service for Kubernetes (ACK) and Alibaba Cloud Elastic Compute Service (ECS). It explains how to deploy:

• One worker for ACK Managed Cluster
• CAA on that Kubernetes cluster
• An Nginx pod backed by CAA pod VM on ECS

Note: Run the following commands from the following directory - src/cloud-api-adaptor

Note: Now Confidential Computing instances are available in some regions.

Note: Official document from Alibaba Cloud can be found here.

Prerequisites

Install Required Tools:

• Install aliyun CLI tool and configure credentials,
• Have an aliyun OSS storage with a bucket.

Create pod VM Image

Note: There is a pre-built Community Image (id:m-2ze1w9aj2aonwckv64cw) for version 0.13.0 in cn-beijing that you can use for testing.

If you want to build a pod VM image yourself, please follow the steps.

1. Create pod VM image.

  PODVM_DISTRO=alinux \
  CLOUD_PROVIDER=alibabacloud \
  IMAGE_URL=https://alinux3.oss-cn-hangzhou.aliyuncs.com/aliyun_3_x64_20G_nocloud_alibase_20250117.qcow2 \
  make podvm-builder podvm-binaries podvm-image

The built image will be available in the root path of following newly built docker image: quay.io/confidential-containers/podvm-alibabacloud-alinux-amd64:<sha256> with name like podvm-*.qcow2. You need to export it from the container image.

2. Upload to OSS storage and create ECS Image.

You will then need to upload the Pod VM image to OSS (Object Storage Service).

export REGION_ID=<region-id>
export IMAGE_FILE=<path-to-qcow2-file>
export BUCKET=<OSS-bucket-name>
export OBJECT=<object-name>

aliyun oss cp ${IMAGE_FILE} oss://${BUCKET}/${OBJECT}

Then, mark the image file as an ECS Image

export IMAGE_NAME=$(basename ${IMAGE_FILE%.*})
aliyun ecs ImportImage --ImageName ${IMAGE_NAME} \
    --region ${REGION_ID} --RegionId ${REGION_ID}
    --BootMode UEFI \
    --DiskDeviceMapping.1.OSSBucket ${BUCKET} --DiskDeviceMapping.1.OSSObject ${OBJECT} \
    --Features.NvmeSupport supported \
    --method POST --force

export POD_IMAGE_ID=<ImageId>

Build CAA development image

If you want to build CAA DaemonSet image yourself:

export registry=<registry-address>
export RELEASE_BUILD=true
export CLOUD_PROVIDER=alibabacloud
make image

After that you should take note of the tag used for this image, we will use it later.

Deploy Kubernetes using ACK Managed Cluster

1. Create ACK Managed Cluster.

  export CONTAINER_CIDR=172.18.0.0/16
  export REGION_ID=cn-beijing
  export ZONES='["cn-beijing-i"]'

  aliyun cs CreateCluster --header "Content-Type=application/json" --body "
  {
    \"cluster_type\":\"ManagedKubernetes\",
    \"name\":\"caa\",
    \"region_id\":\"${REGION_ID}\",
    \"zone_ids\":${ZONES},
    \"enable_rrsa\":true,
    \"container_cidr\":\"${CONTAINER_CIDR}\",
    \"addons\":[
      {
        \"name\":\"flannel\"
      }
    ]
  }"

  export CLUSTER_ID=<cluster-id>
  export SECURITY_GROUP_ID=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r  ".security_group_id")

Wait for the cluster to be created. Get the vSwitch id of the cluster. Then add one worker node to the cluster.

2. Add Internet access for the cluster VPC

  export VPC_ID=$(aliyun cs DescribeClusterDetail --ClusterId ${CLUSTER_ID} | jq -r ".vpc_id")
  export VSWITCH_ID=$(echo ${VSWITCH_IDS} | sed 's/[][]//g' | sed 's/"//g')
  aliyun vpc CreateNatGateway \
    --region ${REGION_ID} \
    --RegionId ${REGION_ID} \
    --VpcId ${VPC_ID} \
    --NatType Enhanced \
    --VSwitchId ${VSWITCH_ID} \
    --NetworkType internet

  export GATEWAY_ID="<NatGatewayId>"
  export SNAT_TABLE_ID="<SnatTableId>"

  # The band width of the public ip (Mbps)
  export BAND_WIDTH=5
  aliyun vpc AllocateEipAddress \
    --region ${REGION_ID} \
    --RegionId ${REGION_ID} \
    --Bandwidth ${BAND_WIDTH}

  export EIP_ID="<AllocationId>"
  export EIP_ADDRESS="<EipAddress>"

  aliyun vpc AssociateEipAddress \
    --region ${REGION_ID} \
    --RegionId ${REGION_ID} \
    --AllocationId ${EIP_ID} \
    --InstanceId ${GATEWAY_ID} \
    --InstanceType Nat

  aliyun vpc CreateSnatEntry \
    --region ${REGION_ID} \
    --RegionId ${REGION_ID} \
    --SnatTableId ${SNAT_TABLE_ID} \
    --SourceVSwitchId ${VSWITCH_ID} \
    --SnatIp ${EIP_ADDRESS}

3. Grant role permissions Give role permission to the cluster to allow the worker to create ECS instances.

Deploy the CAA Helm Chart

Download the CAA Helm deployment artifacts

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_VERSION="0.17.0"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Alibaba Cloud.

export IMAGEID="m-2zef6zaa0j0qz3sunhjp"

Note: Alibaba Cloud builds the images ahead of time. Different regions has different image id to use.

region	IMAGEID
cn-beijing	m-2zef6zaa0j0qz3sunhjp
ap-southeast-1	m-t4n9ocuen5sy6rhbxbk1

Export CAA container image path

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

export CAA_TAG=""

Select peer-pods machine type

export PODVM_INSTANCE_TYPE="ecs.g8i.xlarge"
export DISABLECVM="false"

Note: See the official document for more instance types that supports confidential computing.

Populate the providers/alibabacloud.yaml file

List of all available configuration options can be found in two places:

• Main charts values
• Alibaba Cloud specific values

Run the following command to update the providers/alibabacloud.yaml file:

cat <<EOF > providers/alibabacloud.yaml
provider: alibabacloud
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   alibabacloud:
      IMAGEID: "${IMAGEID}"
      REGION: "${REGION_ID}"
      SECURITY_GROUP_IDS: "${SECURITY_GROUP_ID}"
      VSWITCH_ID: "${VSWITCH_ID}"
      DISABLECVM: ${DISABLECVM}
alibabacloud:
  rrsa:
    enable: true
EOF

Note: If you are not using RRSA for auth, please change the alibabacloud.rrsa.enable to false in the yaml.

Deploy helm chart on the Kubernetes cluster

1. Create namespace managed by Helm:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
    name: confidential-containers-system
    labels:
      app.kubernetes.io/managed-by: Helm
    annotations:
      meta.helm.sh/release-name: peerpods
      meta.helm.sh/release-namespace: confidential-containers-system
   EOF
   

2. Create the secret using kubectl:

   See providers/alibabacloud-secrets.yaml.template for required keys.

   Note: Below example assumes that you are using RRSA for auth hence ALIBABACLOUD_ACCESS_KEY_ID and ALIBABACLOUD_ACCESS_KEY_SECRET are not provided, while ALIBABA_CLOUD_ROLE_ARN and ALIBABA_CLOUD_OIDC_PROVIDER_ARN are provided.

   kubectl create secret generic my-provider-creds \
   -n confidential-containers-system \
   --from-literal=ALIBABA_CLOUD_ROLE_ARN=${ALIBABA_CLOUD_ROLE_ARN} \
   --from-literal=ALIBABA_CLOUD_OIDC_PROVIDER_ARN=${ALIBABA_CLOUD_OIDC_PROVIDER_ARN} \
   --from-literal=ALIBABA_CLOUD_OIDC_TOKEN_FILE=/var/run/secrets/ack.alibabacloud.com/rrsa-tokens/token
   

3. Install helm chart:

   Below command uses customization options -f and --set which are described here.

   helm install peerpods . \
     -f providers/alibabacloud.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --dependency-update \
     -n confidential-containers-system
   

Generic Peer pods Helm charts deployment instructions are also described here.

Run sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying CAA:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

Create an nginx deployment:

echo '
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  runtimeClassName: kata-remote
  containers:
  - name: nginx
    image: registry.openanolis.cn/openanolis/nginx:1.14.1-8.6
' | kubectl apply -f -

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the peer-pod VM was created by running the following command:

aliyun ecs DescribeInstances --RegionId ${REGION_ID} --InstanceName 'podvm-*'

Here you should see the VM associated with the pod nginx. If you run into problems then check the troubleshooting guide here.

Attestation

TODO

Cleanup

Delete all running pods using the runtimeClass kata-remote.

Verify that all peer-pod VMs are deleted. You can use the following command to list all the peer-pod VMs (VMs having prefix podvm) and status:

aliyun ecs DescribeInstances --RegionId ${REGION_ID} --InstanceName 'podvm-*'

Delete the ACK cluster by running the following command:

aliyun cs DELETE /clusters/${CLUSTER_ID} --region ${REGION_ID} --keep_slb false --retain_all_resources false --header "Content-Type=application/json;" --body "{}"

2 - AWS

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on AWS Elastic Kubernetes Service (EKS). It explains how to deploy:

• A single worker node Kubernetes cluster using Elastic Kubernetes Service (EKS)
• CAA on that Kubernetes cluster
• An Nginx pod backed by CAA pod VM

Pre-requisites

Install Required Tools:

• Install kubectl,
• Install Helm,
• Install aws CLI tool,
• Install eksctl CLI tool,
• Ensure that the tools curl, git and jq are installed.

AWS Preparation

• Set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or AWS_PROFILE) and AWS_REGION for AWS CLI access

• Set the region:

• AMD SEV-SNP
• Non-Confidential

export AWS_REGION="us-east-2"

export AWS_REGION="us-east-2"

Deploy Kubernetes using EKS

Make changes to the following environment variable as you see fit:

export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
export CLUSTER_NODE_TYPE="m5.xlarge"
export CLUSTER_NODE_FAMILY_TYPE="Ubuntu2204"
export SSH_KEY=~/.ssh/id_rsa.pub

Example EKS cluster creation using the default AWS VPC-CNI

eksctl create cluster --name "$CLUSTER_NAME" \
    --node-type "$CLUSTER_NODE_TYPE" \
    --node-ami-family "$CLUSTER_NODE_FAMILY_TYPE" \
    --nodes 1 \
    --nodes-min 0 \
    --nodes-max 2 \
    --node-private-networking \
    --kubeconfig "$CLUSTER_NAME"-kubeconfig

Wait for the cluster to be created.

Label the cluster nodes with node.kubernetes.io/worker=

for NODE_NAME in $(kubectl get nodes -o jsonpath='{.items[*].metadata.name}'); do
  kubectl label node $NODE_NAME node.kubernetes.io/worker=
done

Allow required network ports

EKS_VPC_ID=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
--query "cluster.resourcesVpcConfig.vpcId" \
--output text)
echo $EKS_VPC_ID

EKS_CLUSTER_SG=$(aws eks describe-cluster --name "$CLUSTER_NAME" \
  --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" \
  --output text)
echo $EKS_CLUSTER_SG

EKS_VPC_CIDR=$(aws ec2 describe-vpcs --vpc-ids "$EKS_VPC_ID" \
--query 'Vpcs[0].CidrBlock' --output text)
echo $EKS_VPC_CIDR

# agent-protocol-forwarder port
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 15150 --cidr "$EKS_VPC_CIDR"

# vxlan port
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol tcp --port 9000 --cidr "$EKS_VPC_CIDR"
aws ec2 authorize-security-group-ingress --group-id "$EKS_CLUSTER_SG" --protocol udp --port 9000 --cidr "$EKS_VPC_CIDR"

Note:

• Port 15150 is the default port for CAA to connect to the agent-protocol-forwarder running inside the pod VM.
• Port 9000 is the VXLAN port used by CAA. Ensure it doesn’t conflict with the VXLAN port used by the Kubernetes CNI.

Deploy the CAA Helm chart

Download the CAA Helm deployment artifacts

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_VERSION="0.17.0"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in AWS.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export PODVM_AMI_ID=$(aws ec2 describe-images \
    --filters Name=name,Values="podvm-fedora-amd64-${CAA_VERSION//./-}" \
    --query 'Images[*].[ImageId]' --output text)

echo $PODVM_AMI_ID

Export CAA container image path

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

export CAA_TAG=""

Select peer-pods machine type

• AMD SEV-SNP
• Non-Confidential

export PODVM_INSTANCE_TYPE="m6a.large"
export DISABLECVM="false"

export PODVM_INSTANCE_TYPE="t3.large"
export DISABLECVM="true"

Populate the providers/aws.yaml file

List of all available configuration options can be found in two places:

• Main charts values
• AWS specific values

Run the following command to update the providers/aws.yaml file:

cat <<EOF > providers/aws.yaml
provider: aws
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   aws:
      DISABLECVM: ${DISABLECVM}
      PODVM_AMI_ID: "${PODVM_AMI_ID}"
      PODVM_INSTANCE_TYPE: "${PODVM_INSTANCE_TYPE}"
      VXLAN_PORT: 9000
EOF

Deploy helm chart on the Kubernetes cluster

1. Create namespace managed by Helm:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
    name: confidential-containers-system
    labels:
      app.kubernetes.io/managed-by: Helm
    annotations:
      meta.helm.sh/release-name: peerpods
      meta.helm.sh/release-namespace: confidential-containers-system
   EOF
   

2. Create the secret using kubectl:

   See providers/aws-secrets.yaml.template for required keys.

   kubectl create secret generic my-provider-creds \
   -n confidential-containers-system \
   --from-literal=AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
   --from-literal=AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
   --from-file=id_rsa.pub=${SSH_KEY}
   

   Note: --from-file=id_rsa.pub=${SSH_KEY} is optional. It allows user to SSH into the pod VMs for troubleshooting purposes. This option works only for custom debug enabled pod VM images. The prebuilt pod VM images do not have SSH connection enabled.

3. Install helm chart:

   Below command uses customization options -f and --set which are described here.

   helm install peerpods . \
     -f providers/aws.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --dependency-update \
     -n confidential-containers-system
   

Generic Peer pods Helm charts deployment instructions are also described here.

Run sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying CAA:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

Create an nginx deployment:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always
EOF

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the peer pod VM was created by running the following command:

aws ec2 describe-instances --filters "Name=tag:Name,Values=podvm*" \
   --query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value | [0]]' --output table

Here you should see the VM associated with the pod nginx.

Note: If you run into problems then check the troubleshooting guide here.

Cleanup

Delete all running pods using the runtimeclass kata-remote. You can use the following command for the same:

kubectl get pods -A -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,RUNTIMECLASS:.spec.runtimeClassName' | grep kata-remote | awk '{print $1, $2}'

Verify that all peer-pod VMs are deleted. You can use the following command to list all the peer-pod VMs (VMs having prefix podvm) and status:

aws ec2 describe-instances --filters "Name=tag:Name,Values=podvm*" \
--query 'Reservations[*].Instances[*].[InstanceId, Tags[?Key==`Name`].Value | [0], State.Name]' --output table

Delete the EKS cluster by running the following command:

eksctl delete cluster --name=$CLUSTER_NAME 

3 - Azure

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on Azure Kubernetes Service (AKS). It explains how to deploy:

• A single worker node Kubernetes cluster using Azure Kubernetes Service (AKS)
• CAA on that Kubernetes cluster
• An Nginx pod backed by CAA pod VM

Confidential Containers also supports using Azure Key Vault as a resource backend for Trustee. More info

Pre-requisites

Install Required Tools:

• Install kubectl,
• Install Helm,
• Install az CLI tool,
• Ensure that the tools curl, git, jq and sipcalc are installed.

Azure Preparation

Azure login

There are a bunch of steps that require you to be logged into your Azure account:

az login

Retrieve your subscription ID:

export AZURE_SUBSCRIPTION_ID=$(az account show --query id --output tsv)

Set the region:

• AMD SEV-SNP
• Intel TDX
• Non-Confidential

export AZURE_REGION="eastus"

export AZURE_REGION="eastus2"

export AZURE_REGION="eastus"

Resource group

Note: Skip this step if you already have a resource group you want to use. Please, export the resource group name in the AZURE_RESOURCE_GROUP environment variable.

Create an Azure resource group by running the following command:

export AZURE_RESOURCE_GROUP="caa-rg-$(date '+%Y%m%b%d%H%M%S')"

az group create \
  --name "${AZURE_RESOURCE_GROUP}" \
  --location "${AZURE_REGION}"

Deploy Kubernetes using AKS

Make changes to the following environment variable as you see fit:

export CLUSTER_NAME="caa-$(date '+%Y%m%b%d%H%M%S')"
export AKS_WORKER_USER_NAME="azuser"
export AKS_RG="${AZURE_RESOURCE_GROUP}-aks"
export SSH_KEY=~/.ssh/id_rsa.pub

Note: Optionally, deploy the worker nodes into an existing Azure Virtual Network (VNet) and subnet by adding the following flag: --vnet-subnet-id $MY_SUBNET_ID.

Deploy AKS with single worker node to the same resource group you created earlier:

az aks create \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --node-resource-group "${AKS_RG}" \
  --name "${CLUSTER_NAME}" \
  --enable-oidc-issuer \
  --enable-workload-identity \
  --location "${AZURE_REGION}" \
  --node-count 1 \
  --node-vm-size Standard_F4s_v2 \
  --nodepool-labels node.kubernetes.io/worker= \
  --ssh-access disabled \
  --admin-username "${AKS_WORKER_USER_NAME}" \
  --os-sku Ubuntu

Download kubeconfig locally to access the cluster using kubectl:

az aks get-credentials \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --name "${CLUSTER_NAME}"

User assigned identity and federated credentials

CAA needs privileges to talk to Azure API. This privilege is granted to CAA by associating a workload identity to the CAA service account. This workload identity (a.k.a. user assigned identity) is given permissions to create VMs, fetch images and join networks in the next step.

Note: If you use an existing AKS cluster it might need to be configured to support workload identity and OpenID Connect (OIDC), please refer to the instructions in this guide.

Start by creating an identity for CAA:

export AZURE_WORKLOAD_IDENTITY_NAME="${CLUSTER_NAME}-identity"

az identity create \
  --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --location "${AZURE_REGION}"

export USER_ASSIGNED_CLIENT_ID="$(az identity show \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
  --query 'clientId' \
  -otsv)"

Networking

The VMs that will host Pods will commonly require access to internet services, e.g. to pull images from a public OCI registry. A discrete subnet can be created next to the AKS cluster subnet in the same VNet. We then attach a NAT gateway with a public IP to that subnet:

export AZURE_VNET_NAME="$(az network vnet list -g ${AKS_RG} --query '[].name' -o tsv)"
export AKS_CIDR="$(az network vnet show -n $AZURE_VNET_NAME -g $AKS_RG --query "subnets[?name == 'aks-subnet'].addressPrefix" -o tsv)"
# 10.224.0.0/16
export MASK="${AKS_CIDR#*/}"
# 16
PEERPOD_CIDR="$(sipcalc $AKS_CIDR -n 2 | grep ^Network | grep -v current | cut -d' ' -f2)/${MASK}"
# 10.225.0.0/16
az network public-ip create -g "$AKS_RG" -n peerpod
az network nat gateway create -g "$AKS_RG" -l "$AZURE_REGION" --public-ip-addresses peerpod -n peerpod
az network vnet subnet create -g "$AKS_RG" --vnet-name "$AZURE_VNET_NAME" --nat-gateway peerpod --address-prefixes "$PEERPOD_CIDR" -n peerpod
export AZURE_SUBNET_ID="$(az network vnet subnet show -g "$AKS_RG" --vnet-name "$AZURE_VNET_NAME" -n peerpod --query id -o tsv)"

AKS resource group permissions

For CAA to be able to manage VMs assign the identity VM and Network contributor roles, privileges to spawn VMs in $AZURE_RESOURCE_GROUP and attach to a VNet in $AKS_RG.

az role assignment create \
  --role "Virtual Machine Contributor" \
  --assignee "$USER_ASSIGNED_CLIENT_ID" \
  --scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"

az role assignment create \
  --role "Reader" \
  --assignee "$USER_ASSIGNED_CLIENT_ID" \
  --scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${AZURE_RESOURCE_GROUP}"

az role assignment create \
  --role "Network Contributor" \
  --assignee "$USER_ASSIGNED_CLIENT_ID" \
  --scope "/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourcegroups/${AKS_RG}"

Create the federated credential for the CAA ServiceAccount using the OIDC endpoint from the AKS cluster:

export AKS_OIDC_ISSUER="$(az aks show \
  --name "${CLUSTER_NAME}" \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --query "oidcIssuerProfile.issuerUrl" \
  -otsv)"

az identity federated-credential create \
  --name "${CLUSTER_NAME}-federated" \
  --identity-name "${AZURE_WORKLOAD_IDENTITY_NAME}" \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --issuer "${AKS_OIDC_ISSUER}" \
  --subject system:serviceaccount:confidential-containers-system:cloud-api-adaptor \
  --audience api://AzureADTokenExchange

Deploy the CAA Helm chart

Note: If you are using Calico Container Network Interface (CNI) on the Kubernetes cluster, then, configure Virtual Extensible LAN (VXLAN) encapsulation for all inter workload traffic.

Download the CAA Helm deployment artifacts

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_VERSION="0.17.0"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Azure.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export AZURE_IMAGE_ID="/CommunityGalleries/cococommunity-42d8482d-92cd-415b-b332-7648bd978eff/Images/peerpod-podvm-fedora/Versions/${CAA_VERSION}"

SUCCESS_TIME=$(curl -s \
  -H "Accept: application/vnd.github+json" \
  "https://api.github.com/repos/confidential-containers/cloud-api-adaptor/actions/workflows/azure-nightly-build.yml/runs?status=success" \
  | jq -r '.workflow_runs[0].updated_at')

export AZURE_IMAGE_ID="/CommunityGalleries/cocopodvm-d0e4f35f-5530-4b9c-8596-112487cdea85/Images/podvm_image0/Versions/$(date -u -jf "%Y-%m-%dT%H:%M:%SZ" "$SUCCESS_TIME" "+%Y.%m.%d" 2>/dev/null || date -d "$SUCCESS_TIME" +%Y.%m.%d)"

Export CAA container image path

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

export CAA_TAG=""

Select peer-pods machine type

• AMD SEV-SNP
• Intel TDX
• Non-Confidential

export AZURE_INSTANCE_SIZE="Standard_DC2as_v5"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_DC2es_v6"
export DISABLECVM="false"

export AZURE_INSTANCE_SIZE="Standard_D2as_v5"
export DISABLECVM="true"

Populate the providers/azure.yaml file

List of all available configuration options can be found in two places:

• Main charts values
• Azure specific values

Run the following command to update the providers/azure.yaml file:

cat <<EOF > providers/azure.yaml
provider: azure
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
   azure:
      AZURE_IMAGE_ID: "${AZURE_IMAGE_ID}"
      AZURE_REGION: "${AZURE_REGION}"
      AZURE_RESOURCE_GROUP: "${AZURE_RESOURCE_GROUP}"
      AZURE_SUBNET_ID: "${AZURE_SUBNET_ID}"
      AZURE_SUBSCRIPTION_ID: "${AZURE_SUBSCRIPTION_ID}"
      AZURE_INSTANCE_SIZE: "${AZURE_INSTANCE_SIZE}"
      DISABLECVM: ${DISABLECVM}
EOF

Deploy helm chart on the Kubernetes cluster

1. Create namespace managed by Helm:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
    name: confidential-containers-system
    labels:
      app.kubernetes.io/managed-by: Helm
    annotations:
      meta.helm.sh/release-name: peerpods
      meta.helm.sh/release-namespace: confidential-containers-system
   EOF
   

2. Create the secret using kubectl:

   See providers/azure-secrets.yaml.template for required keys.

   Note: Below example assumes that you are using workload identity for authentication hence AZURE_CLIENT_SECRET and AZURE_TENANT_ID are not provided.

   kubectl create secret generic my-provider-creds \
   -n confidential-containers-system \
   --from-literal=AZURE_CLIENT_ID="${USER_ASSIGNED_CLIENT_ID}" \
   --from-file=id_rsa.pub=${SSH_KEY}
   

   Note: --from-file=id_rsa.pub=${SSH_KEY} is optional. It allows user to SSH into the pod VMs for troubleshooting purposes. This option works only for custom debug enabled pod VM images. The prebuilt pod VM images do not have SSH connection enabled.

3. Install helm chart:

   Below command uses customization options -f and --set which are described here.

   helm install peerpods . \
     -f providers/azure.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' \
     --dependency-update \
     -n confidential-containers-system
   

   Note: Above example assumes that you are using workload identity for authentication.
   This line: --set-json daemonset.podLabels='{"azure.workload.identity/use":"true"}' is required only when using workload identity.

Generic Peer pods Helm charts deployment instructions are also described here.

Run sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying Peer Pods Helm Charts:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

Create an nginx deployment:

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always
EOF

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the peer pod VM was created by running the following command:

az vm list \
  --resource-group "${AZURE_RESOURCE_GROUP}" \
  --output table

Here you should see the VM associated with the pod nginx.

Note: If you run into problems then check the troubleshooting guide here.

Pod VM reference values

As part of a Pod VM image build expected PCR measurements are published into an OCI registry.

Pre-requisites

Install ORAS tool to pull the reference values from the OCI registry and GitHub CLI to verify its build provenance.

Verify build provenance

Assert that the measurements have been generated by a trusted build process on the official repository. Specify --format=json to get more details about the build process.

CAA_REPO="confidential-containers/cloud-api-adaptor"
OCI_REGISTRY="ghcr.io/${CAA_REPO}/measurements/azure/podvm:${CAA_VERSION}"
gh attestation verify -R "$CAA_REPO" "oci://${OCI_REGISTRY}"

Retrieve reference values

The PCR values can be used in remote attestation policies to assert the integrity of the PodVM image.

oras pull "$OCI_REGISTRY"
jq -r .measurements.sha256.pcr11 < measurements.json
0x58e8afdf5b105fc6b202eb8e537a9f1512a4b33cd5921171b518645a86ca5a75

Cleanup

If you wish to clean up the whole set up, you can delete the resource group by running the following command:

az group delete \
  --name "${AZURE_RESOURCE_GROUP}" \
  --yes --no-wait

4 - Container Launch with SNP Memory Encryption

Container Launch With Memory Encryption

Launch a Confidential Service

To launch a container with SNP memory encryption, the SNP runtime class (kata-qemu-snp) must be specified. A base alpine docker container (Dockerfile) has been previously built for testing purposes. This image has also been prepared with SSH access and provisioned with a SSH public key for validation purposes.

Here is a sample service yaml specifying the SNP runtime class:

kind: Service
apiVersion: v1
metadata:
  name: "confidential-unencrypted"
spec:
  selector:
    app: "confidential-unencrypted"
  ports:
  - port: 22
---
kind: Deployment
apiVersion: apps/v1
metadata:
  name: "confidential-unencrypted"
spec:
  selector:
    matchLabels:
      app: "confidential-unencrypted"
  template:
    metadata:
      labels:
        app: "confidential-unencrypted"
      annotations:
        io.containerd.cri.runtime-handler: kata-qemu-snp
    spec:
      runtimeClassName: kata-qemu-snp
      containers:
      - name: "confidential-unencrypted"
        image: ghcr.io/kata-containers/test-images:unencrypted-nightly
        imagePullPolicy: Always

Save the contents of this yaml to a file called confidential-unencrypted.yaml.

Start the service:

kubectl apply -f confidential-unencrypted.yaml

Check for errors:

kubectl describe pod confidential-unencrypted

If there are no errors in the Events section, then the container has been successfully created with SNP memory encryption.

Validate SNP Memory Encryption

The container dmesg log can be parsed to indicate that SNP memory encryption is enabled and active. The container image defined in the yaml sample above was built with a predefined key that is authorized for SSH access.

Get the pod IP:

pod_ip=$(kubectl get pod -o wide | grep confidential-unencrypted | awk '{print $6;}')

Download and save the SSH private key and set the permissions.

wget https://github.com/kata-containers/kata-containers/raw/main/tests/integration/kubernetes/runtimeclass_workloads/confidential/unencrypted/ssh/unencrypted -O confidential-image-ssh-key

chmod 600 confidential-image-ssh-key

The following command will run a remote SSH command on the container to check if SNP memory encryption is active:

ssh -i confidential-image-ssh-key \
  -o "StrictHostKeyChecking no" \
  -t root@${pod_ip} \
  'dmesg | grep "Memory Encryption Features""'

If SNP is enabled and active, the output should return:

[    0.150045] Memory Encryption Features active: AMD SNP

5 - GCP

This documentation will walk you through setting up CAA (a.k.a. Peer Pods) on Google Kubernetes Engine (GKE). It explains how to deploy:

• A single worker node Kubernetes cluster using GKE
• CAA on that Kubernetes cluster
• A sample application backed by a CAA pod VM

Pre-requisites

Install Required Tools:

• Install kubectl,
• Install Helm,
• Install gcloud CLI tool.

Google Cloud Project:

• Ensure you have a Google Cloud project created,
• Note the Project ID (export it as GCP_PROJECT_ID).

GCP Preparation

Start by authenticating with Google and choosing your project:

export GCP_PROJECT_ID="YOUR_PROJECT_ID"
gcloud auth login
gcloud config set project ${GCP_PROJECT_ID}

Enable the necessary API:

gcloud services enable container.googleapis.com --project=${GCP_PROJECT_ID}

Create a service account with the necessary permissions:

gcloud iam service-accounts create peerpods \
  --description="Peerpods Service Account" \
  --display-name="Peerpods Service Account"

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:peerpods@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/compute.instanceAdmin.v1"

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:peerpods@${GCP_PROJECT_ID}.iam.gserviceaccount.com" \
  --role="roles/iam.serviceAccountUser"

Generate and save the credentials file:

gcloud iam service-accounts keys create \
  ~/.config/gcloud/peerpods_application_key.json \
  --iam-account=peerpods@${GCP_PROJECT_ID}.iam.gserviceaccount.com

export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/peerpods_application_key.json

Configure additional environment variables that will be used later.

Set the region:

export GCP_REGION="us-central1"

Set the PodVM instance type:

• AMD SEV-SNP
• Intel TDX
• Non-Confidential

export PODVM_INSTANCE_TYPE="n2d-standard-4"
export DISABLECVM=false
export GCP_CONFIDENTIAL_TYPE="SEV" # SEV or SEV_SNP
export GCP_DISK_TYPE="pd-standard"

export PODVM_INSTANCE_TYPE="c3-standard-4"
export DISABLECVM=false
export GCP_CONFIDENTIAL_TYPE="TDX"
export GCP_DISK_TYPE="pd-balanced"

export PODVM_INSTANCE_TYPE="e2-medium"
export DISABLECVM=true
export GCP_CONFIDENTIAL_TYPE=""
export GCP_DISK_TYPE="pd-standard"

Deploy Kubernetes Using GKE

Deploy a single node Kubernetes cluster using GKE:

gcloud container clusters create my-cluster \
  --zone ${GCP_REGION}-a \
  --machine-type "e2-standard-4" \
  --image-type UBUNTU_CONTAINERD \
  --num-nodes 1

Label the worker nodes:

kubectl get nodes --selector='!node-role.kubernetes.io/master' -o name | \
xargs -I{} kubectl label {} node.kubernetes.io/worker=

Configure VPC network

We need to make sure port 15150 is open under the default VPC network:

gcloud compute firewall-rules create allow-port-15150 \
    --project=${GCP_PROJECT_ID} \
    --network=default \
    --allow=tcp:15150

For production scenarios, it is advisable to restrict the source IP range to minimize security risks. For example, you can restrict the source range to a specific IP address or CIDR block:

gcloud compute firewall-rules create allow-port-15150-restricted \
   --project=${GCP_PROJECT_ID} \
   --network=default \
   --allow=tcp:15150 \
   --source-ranges=[YOUR_EXTERNAL_IP]

Deploy the CAA Helm chart

Download the CAA Helm deployment artifacts

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_VERSION="0.17.0"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/tags/v${CAA_VERSION}.tar.gz"
tar -xvzf "v${CAA_VERSION}.tar.gz"
cd "cloud-api-adaptor-${CAA_VERSION}/src/cloud-api-adaptor/install/charts/peerpods"

export CAA_BRANCH="main"
curl -LO "https://github.com/confidential-containers/cloud-api-adaptor/archive/refs/heads/${CAA_BRANCH}.tar.gz"
tar -xvzf "${CAA_BRANCH}.tar.gz"
cd "cloud-api-adaptor-${CAA_BRANCH}/src/cloud-api-adaptor/install/charts/peerpods"

Export PodVM image version

Exports the PodVM image ID used by peer pods. This variable tells the deployment tooling which PodVM image version to use when creating peer pod virtual machines in Google Cloud.

The image is pulled from the Coco community gallery (or manually built) and must match the current CAA release version.

• Versions:
• Last Release
• Latest Build
• DIY

export PODVM_IMAGE_ID="/projects/it-cloud-gcp-prod-osc-devel/global/images/fedora-mkosi-tee-amd-1-11-0"

Export CAA container image path

Define the Cloud API Adaptor (CAA) container image to deploy. These variables tell the deployment tooling which CAA image and architecture-specific tag to pull and run. The tag is derived from the CAA release version to ensure compatibility with the selected PodVM image and configuration.

• Versions:
• Last Release
• Latest Build
• DIY

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"
export CAA_TAG="v${CAA_VERSION}-amd64"

export CAA_IMAGE="quay.io/confidential-containers/cloud-api-adaptor"

export CAA_TAG=""

Populate the providers/gcp.yaml file

List of all available configuration options can be found in two places:

• Main charts values
• GCP specific values

Run the following command to update the providers/gcp.yaml file:

cat <<EOF > providers/gcp.yaml
provider: gcp
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
  gcp:
    GCP_NETWORK: "global/networks/default"
    GCP_PROJECT_ID: "${GCP_PROJECT_ID}"
    GCP_ZONE: "${GCP_REGION}-a"
    GCP_MACHINE_TYPE: "${PODVM_INSTANCE_TYPE}"
    GCP_DISK_TYPE: "${GCP_DISK_TYPE}"
    PODVM_IMAGE_NAME: "${PODVM_IMAGE_ID}"
    GCP_CONFIDENTIAL_TYPE: "${GCP_CONFIDENTIAL_TYPE}"
    DISABLECVM: ${DISABLECVM}
EOF

Deploy helm chart on the Kubernetes cluster

1. Create namespace managed by Helm:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: Namespace
   metadata:
    name: confidential-containers-system
    labels:
      app.kubernetes.io/managed-by: Helm
    annotations:
      meta.helm.sh/release-name: peerpods
      meta.helm.sh/release-namespace: confidential-containers-system
   EOF
   

2. Create the secret using kubectl:

   See providers/gcp-secrets.yaml.template for required keys.

   kubectl create secret generic my-provider-creds \
    -n confidential-containers-system \
    --from-file=GCP_CREDENTIALS="${GOOGLE_APPLICATION_CREDENTIALS}"
   

3. Install helm chart:

   Below command uses customization options -f and --set which are described here.

   helm install peerpods . \
     -f providers/gcp.yaml \
     --set secrets.mode=reference \
     --set secrets.existingSecretName=my-provider-creds \
     --dependency-update \
     -n confidential-containers-system
   

Generic Peer pods Helm charts deployment instructions are also described here.

Run a sample application

Ensure runtimeclass is present

Verify that the runtimeclass is created after deploying Peer Pods Helm Charts:

kubectl get runtimeclass

Once you can find a runtimeclass named kata-remote then you can be sure that the deployment was successful. A successful deployment will look like this:

$ kubectl get runtimeclass
NAME          HANDLER       AGE
kata-remote   kata-remote   7m18s

Deploy workload

• CoCo Secret Retrieval
• Basic nginx

# initdata.toml
algorithm = "sha384"
version = "0.1.0"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]
url = 'http://127.0.0.1:8080'

[token_configs.kbs]
url = 'http://127.0.0.1:8080'
cert = """
-----BEGIN CERTIFICATE-----
MIIDljCCAn6gAwIBAgIUR/UNh13GFam4emgludtype/S9BIwDQYJKoZIhvcNAQEL
BQAwdTELMAkGA1UEBhMCQ04xETAPBgNVBAgMCFpoZWppYW5nMREwDwYDVQQHDAhI
YW5nemhvdTERMA8GA1UECgwIQUFTLVRFU1QxFDASBgNVBAsMC0RldmVsb3BtZW50
MRcwFQYDVQQDDA5BQVMtVEVTVC1IVFRQUzAeFw0yNDAzMTgwNzAzNTNaFw0yNTAz
MTgwNzAzNTNaMHUxCzAJBgNVBAYTAkNOMREwDwYDVQQIDAhaaGVqaWFuZzERMA8G
A1UEBwwISGFuZ3pob3UxETAPBgNVBAoMCEFBUy1URVNUMRQwEgYDVQQLDAtEZXZl
bG9wbWVudDEXMBUGA1UEAwwOQUFTLVRFU1QtSFRUUFMwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDfp1aBr6LiNRBlJUcDGcAbcUCPG6UzywtVIc8+comS
ay//gwz2AkDmFVvqwI4bdp/NUCwSC6ShHzxsrCEiagRKtA3af/ckM7hOkb4S6u/5
ewHHFcL6YOUp+NOH5/dSLrFHLjet0dt4LkyNBPe7mKAyCJXfiX3wb25wIBB0Tfa0
p5VoKzwWeDQBx7aX8TKbG6/FZIiOXGZdl24DGARiqE3XifX7DH9iVZ2V2RL9+3WY
05GETNFPKtcrNwTy8St8/HsWVxjAzGFzf75Lbys9Ff3JMDsg9zQzgcJJzYWisxlY
g3CmnbENP0eoHS4WjQlTUyY0mtnOwodo4Vdf8ZOkU4wJAgMBAAGjHjAcMBoGA1Ud
EQQTMBGCCWxvY2FsaG9zdIcEfwAAATANBgkqhkiG9w0BAQsFAAOCAQEAKW32spii
t2JB7C1IvYpJw5mQ5bhIlldE0iB5rwWvNbuDgPrgfTI4xiX5sumdHw+P2+GU9KXF
nWkFRZ9W/26xFrVgGIS/a07aI7xrlp0Oj+1uO91UhCL3HhME/0tPC6z1iaFeZp8Y
T1tLnafqiGiThFUgvg6PKt86enX60vGaTY7sslRlgbDr9sAi/NDSS7U1PviuC6yo
yJi7BDiRSx7KrMGLscQ+AKKo2RF1MLzlJMa1kIZfvKDBXFzRd61K5IjDRQ4HQhwX
DYEbQvoZIkUTc1gBUWDcAUS5ztbJg9LCb9WVtvUTqTP2lGuNymOvdsuXq+sAZh9b
M9QaC1mzQ/OStg==
-----END CERTIFICATE-----
"""
'''

"cdh.toml"  = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = 'http://1.2.3.4:8080'
kbs_cert = """
-----BEGIN CERTIFICATE-----
MIIFTDCCAvugAwIBAgIBADBGBgkqhkiG9w0BAQowOaAPMA0GCWCGSAFlAwQCAgUA
oRwwGgYJKoZIhvcNAQEIMA0GCWCGSAFlAwQCAgUAogMCATCjAwIBATB7MRQwEgYD
VQQLDAtFbmdpbmVlcmluZzELMAkGA1UEBhMCVVMxFDASBgNVBAcMC1NhbnRhIENs
YXJhMQswCQYDVQQIDAJDQTEfMB0GA1UECgwWQWR2YW5jZWQgTWljcm8gRGV2aWNl
czESMBAGA1UEAwwJU0VWLU1pbGFuMB4XDTIzMDEyNDE3NTgyNloXDTMwMDEyNDE3
NTgyNlowejEUMBIGA1UECwwLRW5naW5lZXJpbmcxCzAJBgNVBAYTAlVTMRQwEgYD
VQQHDAtTYW50YSBDbGFyYTELMAkGA1UECAwCQ0ExHzAdBgNVBAoMFkFkdmFuY2Vk
IE1pY3JvIERldmljZXMxETAPBgNVBAMMCFNFVi1WQ0VLMHYwEAYHKoZIzj0CAQYF
K4EEACIDYgAExmG1ZbuoAQK93USRyZQcsyobfbaAEoKEELf/jK39cOVJt1t4s83W
XM3rqIbS7qHUHQw/FGyOvdaEUs5+wwxpCWfDnmJMAQ+ctgZqgDEKh1NqlOuuKcKq
2YAWE5cTH7sHo4IBFjCCARIwEAYJKwYBBAGceAEBBAMCAQAwFwYJKwYBBAGceAEC
BAoWCE1pbGFuLUIwMBEGCisGAQQBnHgBAwEEAwIBAzARBgorBgEEAZx4AQMCBAMC
AQAwEQYKKwYBBAGceAEDBAQDAgEAMBEGCisGAQQBnHgBAwUEAwIBADARBgorBgEE
AZx4AQMGBAMCAQAwEQYKKwYBBAGceAEDBwQDAgEAMBEGCisGAQQBnHgBAwMEAwIB
CDARBgorBgEEAZx4AQMIBAMCAXMwTQYJKwYBBAGceAEEBEDDhCejDUx6+dlvehW5
cmmCWmTLdqI1L/1dGBFdia1HP46MC82aXZKGYSutSq37RCYgWjueT+qCMBE1oXDk
d1JOMEYGCSqGSIb3DQEBCjA5oA8wDQYJYIZIAWUDBAICBQChHDAaBgkqhkiG9w0B
AQgwDQYJYIZIAWUDBAICBQCiAwIBMKMDAgEBA4ICAQACgCai9x8DAWzX/2IelNWm
ituEBSiq9C9eDnBEckQYikAhPasfagnoWFAtKu/ZWTKHi+BMbhKwswBS8W0G1ywi
cUWGlzigI4tdxxf1YBJyCoTSNssSbKmIh5jemBfrvIBo1yEd+e56ZJMdhN8e+xWU
bvovUC2/7Dl76fzAaACLSorZUv5XPJwKXwEOHo7FIcREjoZn+fKjJTnmdXce0LD6
9RHr+r+ceyE79gmK31bI9DYiJoL4LeGdXZ3gMOVDR1OnDos5lOBcV+quJ6JujpgH
d9g3Sa7Du7pusD9Fdap98ocZslRfFjFi//2YdVM4MKbq6IwpYNB+2PCEKNC7SfbO
NgZYJuPZnM/wViES/cP7MZNJ1KUKBI9yh6TmlSsZZOclGJvrOsBZimTXpATjdNMt
cluKwqAUUzYQmU7bf2TMdOXyA9iH5wIpj1kWGE1VuFADTKILkTc6LzLzOWCofLxf
onhTtSDtzIv/uel547GZqq+rVRvmIieEuEvDETwuookfV6qu3D/9KuSr9xiznmEg
xynud/f525jppJMcD/ofbQxUZuGKvb3f3zy+aLxqidoX7gca2Xd9jyUy5Y/83+ZN
bz4PZx81UJzXVI9ABEh8/xilATh1ZxOePTBJjN7lgr0lXtKYjV/43yyxgUYrXNZS
oLSG2dLCK9mjjraPjau34Q==
-----END CERTIFICATE-----
"""
'''

"policy.rego" = '''
package agent_policy

import future.keywords.in
import future.keywords.every

import input

# Default values, returned by OPA when rules cannot be evaluated to true.
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := false
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default OnlineCPUMemRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StatsContainerRequest := true
default TtyWinResizeRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := false
'''

INITDATA=$(cat initdata.toml | gzip | base64 -w0)

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: "$INITDATA"
spec:
  runtimeClassName: kata-remote
  containers:
    - name: example-container
      image: alpine:latest
      command:
        - sleep
        - "3600"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
EOF

kubectl exec -it example-pod -- curl http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1

cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: default
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      runtimeClassName: kata-remote
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80
        imagePullPolicy: Always
EOF

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the PodVM was created by running the following command:

gcloud compute instances list

Here you should see the VM associated with the pod used by the example above.

Cleanup

Delete all running pods using the runtimeclass kata-remote. You can use the following command for the same:

kubectl get pods -A -o custom-columns='NAME:.metadata.name,NAMESPACE:.metadata.namespace,RUNTIMECLASS:.spec.runtimeClassName' | grep kata-remote | awk '{print $1, $2}'

Verify that all peer-pod VMs are deleted. You can use the following command to list all the peer-pod VMs (VMs having prefix podvm) and status:

gcloud compute instances list \
  --filter="name~'podvm.*'" \
  --format="table(name,zone,status)"

Delete the GKE cluster by running the following command:

gcloud container clusters delete my-cluster --zone ${GCP_REGION}-a

Troubleshooting

Note: If your case is not covered in section below check the troubleshooting guide here.

VM Doesn’t Start

Starting with GKE version 1.27, GCP configures containerd with the discard_unpacked_layers=true flag to optimize disk usage by removing compressed image layers after they are unpacked. However, this can cause issues with PeerPods, as the workload may fail to locate required layers. To avoid this, disable the discard_unpacked_layers setting in the containerd configuration.

Most of the time you will see a generic message such as the following:

Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:<SHA>: failed to get reader from content store: content digest sha256:<SHA>: not found

To disable the discard_unpacked_layers setting in the containerd configuration on Google Kubernetes Engine (GKE) version 1.27 or later, follow these steps:

1. SSH to worker node Google console
2. Run command sudo sed -i 's/discard_unpacked_layers = true/discard_unpacked_layers = false/' /etc/containerd/config.toml
3. Verify changed property sudo cat /etc/containerd/config.toml | grep discard_unpacked_layers
4. Restart containerd sudo systemctl restart containerd

6 - NVIDIA confidential NIM deployment

This example adapts an NVIDIA NIM inference deployment on Kubernetes to run with Confidential Containers. This particular scenario targets one AMD SEV-SNP Kubernetes worker node with NVIDIA GPU confidential computing support. The same NIM deployment pattern can be adapted to Intel TDX nodes, but the reference values and attestation policy must be generated for TDX rather than SNP. Those TDX-specific steps are out of scope for this exercise.

NVIDIA NIM is a set of inference microservices that package foundation models as containers with optimized runtimes and HTTP APIs for GPU infrastructure. This example starts with a plain NIM Pod manifest for the nvcr.io/nim/meta/llama-3.1-8b-instruct:1.13.1 image, which serves the Meta Llama 3.1 8B Instruct model through a chat completions API. The optional baseline step runs that manifest with the non-confidential kata-qemu-nvidia-gpu runtime class and queries its health, model list, and chat completion endpoints on port 8000. The confidential scenario uses the kata-qemu-nvidia-gpu-snp runtime class which moves the Pod into a confidential VM, but the change alone is not sufficient: A secure deployment also needs Trustee’s Key Broker Service (KBS), guest pull, Attestation Agent (AA) and Confidential Data Hub (CDH) configuration, sealed secrets, image signature policy, a generated Kata agent policy, trusted storage, and a KBS policy that approves the expected CPU, GPU, and initdata evidence. The checkpoints below add those pieces one at a time.

Prerequisites

Before starting, prepare:

• a Kubernetes cluster with Confidential Containers, Kata, the NVIDIA GPU Operator, and a GPU confidential runtime class such as kata-qemu-nvidia-gpu-snp. This setup can be prepared by following the NVIDIA Confidential Containers deployment guide
• access to the Kubernetes worker node, because this tutorial collects SNP launch inputs from the live QEMU process
• an NGC API key that can pull the NIM image
• kubectl, curl, jq, openssl, oras, skopeo, envsubst, base64, tar, zstd, and cargo installed on the node
• Docker with the Docker Compose plugin installed on the node
• Python packages cryptography and jwcrypto for the sealed-secret helper snippet installed on the node

Recipe Roadmap

The rest of this document builds the confidential deployment in practical checkpoints. Each checkpoint should leave the cluster in a state that can be observed and corrected before moving on.

1. Optional Non-Confidential Baseline: switch the GPU to non-confidential mode, deploy the baseline Pod, show that the NIM API is reachable, remove the Pod, then switch the GPU back to confidential mode.
2. Deploy Trustee and Install kbs-client: deploy Trustee directly on the tutorial node, select matching Trustee artifacts, configure the KBS HTTPS endpoint, and set up the KBS admin helper tool. This keeps the demo self-contained, but a production deployment should run Trustee and its admin operations from a trusted environment.
3. Prepare the TEE NIM Manifest and Sealed Secret: create the sealed runtime secret value, then define the SNP GPU Pod manifest that references it, the registry pull Secret, and trusted storage.
4. Create the Pod Initdata Annotation: configure AA and CDH with the KBS address, KBS certificate, registry credential URI, and image signature policy URI.
5. Generate the Kata Agent Policy: run genpolicy on the TEE Pod manifest, combine the policy with the initdata, attach the generated cc_init_data annotation to the manifest, and keep that generated manifest ready without deploying it. This is normally done in a trusted build or release environment.
6. Collect SNP Launch Reference Values: when the launch reference values are not already available, launch a lightweight measurement Pod with the generated cc_init_data, record the SNP launch measurement, SNP TCB values, and attested initdata digest, then remove the measurement Pod. This observed collection is for the demo; production should approve reference values ahead of deployment.
7. Provision KBS Resources, Reference Values, and Policy: configure NVIDIA image signature verification, NGC registry credentials, the NIM runtime API key, SNP reference values, the sealed-secret verification key, and the KBS resource policy that requires affirming CPU and GPU evidence plus the approved initdata digest.
8. Prepare the Trusted Storage Resources: create the trusted storage dependency referenced by the TEE Pod manifest.
9. Exercise the End-to-End Scenario: create the Kubernetes-side Secret objects, deploy the policy-bearing TEE Pod manifest once, and show that the NIM API is reachable under the KBS policy.

Initial NIM Manifest

Export an NGC API key that can pull the NIM image, then create the initial manifest template. You will only need to deploy this non-confidential manifest if you exercise the optional baseline check in checkpoint 1.

The manifest uses the NGC API key in two places: ngc-secret-instruct lets Kubernetes pull the image from nvcr.io, and ngc-api-key-instruct passes the key into the NIM container at runtime. This recipe uses the same key for both purposes to keep the example small. In practice, you can use two different API keys: one for registry access and one for the container’s runtime logic.

This file is the base manifest for the optional baseline deployment. It keeps ${NGC_API_KEY} as a placeholder; the validation and apply steps below use envsubst to substitute the exported value before passing the manifest to kubectl.

export NGC_API_KEY="<NGC_API_KEY>"

cat <<'EOF' | tee nvidia-nim-llama-3-1-8b-instruct.yaml >/dev/null
apiVersion: v1
kind: Secret
metadata:
  name: ngc-secret-instruct
type: kubernetes.io/dockerconfigjson
stringData:
  .dockerconfigjson: |
    {
      "auths": {
        "nvcr.io": {
          "username": "$oauthtoken",
          "password": "${NGC_API_KEY}"
        }
      }
    }
---
apiVersion: v1
kind: Secret
metadata:
  name: ngc-api-key-instruct
type: Opaque
stringData:
  api-key: "${NGC_API_KEY}"
---
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-nim-llama-3-1-8b-instruct
  labels:
    app: nvidia-nim-llama-3-1-8b-instruct
spec:
  restartPolicy: Never
  runtimeClassName: kata-qemu-nvidia-gpu
  imagePullSecrets:
    - name: ngc-secret-instruct
  containers:
    - name: nvidia-nim-llama-3-1-8b-instruct
      image: nvcr.io/nim/meta/llama-3.1-8b-instruct:1.13.1
      ports:
        - containerPort: 8000
          name: http-openai
      livenessProbe:
        httpGet:
          path: /v1/health/live
          port: http-openai
        initialDelaySeconds: 15
        periodSeconds: 10
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /v1/health/ready
          port: http-openai
        initialDelaySeconds: 15
        periodSeconds: 10
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 3
      startupProbe:
        httpGet:
          path: /v1/health/ready
          port: http-openai
        initialDelaySeconds: 360
        periodSeconds: 10
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 120
      env:
        - name: NGC_API_KEY
          valueFrom:
            secretKeyRef:
              name: ngc-api-key-instruct
              key: api-key
      resources:
        limits:
          nvidia.com/pgpu: "1"
          cpu: "16"
          memory: "48Gi"
EOF

Checkpoint 1: Optional Non-Confidential Baseline

Skip this checkpoint if you want to proceed directly to the confidential deployment. It is only a baseline sanity check for the plain NIM Pod manifest running on Kata Containers. It temporarily puts the GPU in non-confidential mode, runs the non-TEE Pod, then removes the Pod. The checkpoint uses node labels to request each mode change, and the NVIDIA GPU Operator orchestrates the GPU mode switch based on those labels. The last step requests confidential GPU mode again so the remaining checkpoints continue on the confidential deployment path. For more background on the GPU runtime classes and confidential GPU mode labels, see the NVIDIA GPU examples.

Ensure that the node is in VM passthrough mode and non-confidential GPU mode:

kubectl label nodes --all \
  nvidia.com/gpu.workload.config=vm-passthrough \
  --overwrite

kubectl label nodes --all nvidia.com/cc.mode=off --overwrite

Changing the CC mode may restart GPU Operator operands. Wait for the node label to report that the transition completed, then wait for the GPU Operator controllers to settle. Waiting for all current Pods directly can be racy here, because the operator may delete and recreate operand Pods during the mode transition.

kubectl wait \
  --for=jsonpath='{.metadata.labels.nvidia\.com/cc\.mode\.state}'=off \
  node --all \
  --timeout=15m

kubectl -n gpu-operator wait \
  --for=condition=Available \
  deployment --all \
  --timeout=10m

kubectl -n gpu-operator rollout status \
  daemonset/nvidia-vfio-manager \
  --timeout=10m

kubectl -n gpu-operator rollout status \
  daemonset/nvidia-sandbox-validator \
  --timeout=10m

kubectl -n gpu-operator rollout status \
  daemonset/nvidia-kata-sandbox-device-plugin-daemonset \
  --timeout=10m

kubectl -n gpu-operator rollout status \
  daemonset/nvidia-cc-manager \
  --timeout=10m

kubectl get nodes \
  -L nvidia.com/gpu.workload.config,nvidia.com/cc.mode,nvidia.com/cc.mode.state

Deploy the non-confidential manifest:

: "${NGC_API_KEY:?set NGC_API_KEY to your NGC API key}"

envsubst '${NGC_API_KEY}' \
  < nvidia-nim-llama-3-1-8b-instruct.yaml \
  | kubectl apply -f -

kubectl wait --for=condition=Ready \
  --timeout=600s \
  pod/nvidia-nim-llama-3-1-8b-instruct

For the showcase, query the Pod IP directly from a machine that can reach the cluster Pod network, such as the single-node cluster host. This baseline does not create a Kubernetes Service. In production, you would usually expose NIM through a Service, Ingress, Gateway, or another controlled endpoint.

Check that NIM is ready and list the model name:

POD_IP="$(kubectl get pod nvidia-nim-llama-3-1-8b-instruct -o jsonpath='{.status.podIP}')"

curl -fsS "http://${POD_IP}:8000/v1/health/ready" | jq .
curl -fsS "http://${POD_IP}:8000/v1/models" | jq -r '.data[].id'

Send a minimal chat completion request:

MODEL_NAME="$(curl -fsS "http://${POD_IP}:8000/v1/models" | jq -r '.data[0].id')"

curl -fsS "http://${POD_IP}:8000/v1/chat/completions" \
  -H 'Content-Type: application/json' \
  -d "$(jq -n --arg model "${MODEL_NAME}" '{
    model: $model,
    messages: [{role: "user", content: "Reply with exactly: hello from nim"}],
    max_tokens: 16,
    temperature: 0
  }')" | jq -r '.choices[0].message.content'

After the baseline check passes, remove the non-TEE sample Pod and its Kubernetes Secrets:

kubectl delete pod nvidia-nim-llama-3-1-8b-instruct --ignore-not-found
kubectl delete secret ngc-secret-instruct ngc-api-key-instruct \
  --ignore-not-found

Return the node to confidential GPU mode before continuing:

kubectl label nodes --all nvidia.com/cc.mode=on --overwrite

Repeat the same CC mode stabilization checks shown above, but wait for nvidia.com/cc.mode.state to become on instead of off. Confirm the node reports cc.mode=on and cc.mode.state=on before continuing.

Checkpoint 2: Deploy Trustee and Install kbs-client

This checkpoint deploys Trustee directly on the tutorial node and installs kbs-client, the admin tool used later to provision KBS resources, reference values, and resource policy. KBS is the Trustee service that serves the guest-facing attestation and resource APIs, as well as the admin API used by kbs-client. Trustee can be deployed with Docker Compose, Helm charts, Kubernetes manifests, or another secure bootstrap process that matches your environment. This tutorial uses Docker Compose because it starts KBS, the Attestation Service (AS), and the Reference Value Provider Service (RVPS) together from the Trustee source tree. In the Compose deployment, KBS, AS, and RVPS run as separate containers: KBS calls AS to verify evidence, and AS calls RVPS for the reference values used during verification. The Compose file also runs a one-shot setup service that creates demo key material before the long-running services use it. The steps below call out all configuration and identity material consumed or created by that deployment, because production deployments may provide those inputs through separate processes, or replace the demo-generated outputs with material that has been reviewed, signed, or protected separately.

Running Trustee on the same single-node system as the workload is only for this showcase. It simplifies networking and keeps the recipe self-contained, but it does not provide the trust separation that a production deployment needs. In production, deploy Trustee in a separate trusted environment, use durable storage for KBS state, serve the KBS endpoint with a certificate issued by a trust chain that guests are configured to trust, and restrict the admin interface carefully. For alternate deployment methods, see the Trustee installation documentation.

Set KBS Deployment Parameters

Start by defining the local working directory, the KBS URL that guest components will use, and the derived paths used by the Compose deployment. Docker Compose publishes KBS on the worker host. Because this tutorial runs on a single-node cluster, the guest can use the node IP and the published host port:

# Local paths.
KBS_WORKDIR="${HOME}/nim-kbs"
KBS_TRUSTEE_DIR="${KBS_WORKDIR}/trustee"
KBS_CONFIG_DIR="${KBS_TRUSTEE_DIR}/kbs/config"
KBS_COMPOSE_CONFIG_DIR="${KBS_CONFIG_DIR}/docker-compose"
KBS_DATA_DIR="${KBS_TRUSTEE_DIR}/kbs/data"
KBS_STORAGE_DIR="${KBS_DATA_DIR}/kbs-storage"

# Artifact selection.
KBS_CLIENT_ARCH="${KBS_CLIENT_ARCH:-x86_64}"

# Network endpoint.
KBS_PORT="${KBS_PORT:-8080}"
KBS_NODE_IP="$(kubectl get nodes \
  -o jsonpath='{.items[0].status.addresses[?(@.type=="InternalIP")].address}')"
KBS_URL="https://${KBS_NODE_IP}:${KBS_PORT}"

mkdir -p "${KBS_WORKDIR}"

Most paths are derived from KBS_WORKDIR; normally you only choose KBS_WORKDIR, KBS_PORT, and optionally KBS_CLIENT_ARCH:

• KBS_TRUSTEE_DIR holds the downloaded Trustee source tree.
• KBS_CONFIG_DIR is mounted into the Compose services as the demo configuration and identity directory.
• KBS_COMPOSE_CONFIG_DIR contains the KBS TOML configuration used by the Compose deployment.
• KBS_DATA_DIR is the Compose-managed demo state root. RVPS reference values and AS state are stored in sibling directories under this tree.
• KBS_STORAGE_DIR is the KBS resource and resource-policy repository under KBS_DATA_DIR.

The KBS_URL value is later written into the pod’s initdata document so AA and CDH inside the guest know where to reach KBS. With this Compose deployment, KBS listens on the host port selected by KBS_PORT; the Kata guest reaches it through the node IP. In production, the KBS endpoint is usually a stable service address in a separate trusted environment, and guest networking must be designed so that the guest can reach that endpoint.

In production, use storage that matches the Trustee trust boundary, such as a dedicated persistent volume or a trusted external backend.

Select Trustee Deployment Artifacts

The next step selects the Trustee source tree, the Compose service images, and the matching kbs-client artifact. In general, use the Trustee version validated for the CoCo release and guest components in your cluster. The NVIDIA supported-platforms documentation is intended to be the source that tells you which Trustee version to use for a given platform. See the NVIDIA supported platforms page for that version mapping. Kata deployments installed using the kata-deploy Helm chart usually provide release metadata in /opt/kata/versions.yaml. The commands below use the coco-trustee entry in that file to resolve the matching Trustee source reference automatically. If the file is unavailable, inspect the upstream Kata Containers versions.yaml at the Kata tag shipped with your CoCo release and read externals.coco-trustee.version. Under the current release convention, the matching Trustee tag is one minor version behind the CoCo release (for example, CoCo v0.18.0 uses Trustee v0.17.0). If the reference still cannot be resolved, look up the Trustee reference validated for your platform, set KBS_TRUSTEE_REF explicitly, and rerun the commands.

This works well for the tutorial because the commands run on the same node where Kata is installed. For production, treat release selection as part of your deployment process instead. If you already know the Trustee reference to use, set KBS_TRUSTEE_REF before running the snippet and skip the automatic lookup:

KATA_VERSIONS_FILE="${KATA_VERSIONS_FILE:-/opt/kata/versions.yaml}"
KBS_TRUSTEE_REF="${KBS_TRUSTEE_REF:-}"
KBS_IMAGE_REPO="ghcr.io/confidential-containers/staged-images/kbs-grpc-as"
KBS_AS_IMAGE_REPO="ghcr.io/confidential-containers/staged-images/coco-as-grpc"
KBS_RVPS_IMAGE_REPO="ghcr.io/confidential-containers/staged-images/rvps"
KBS_CLIENT_IMAGE_REPO="ghcr.io/confidential-containers/staged-images/kbs-client"

if test -z "${KBS_TRUSTEE_REF}"; then
  if ! test -r "${KATA_VERSIONS_FILE}"; then
    echo "ERROR: cannot resolve KBS_TRUSTEE_REF because ${KATA_VERSIONS_FILE} is not readable." >&2
    echo "Look up the validated Trustee reference for your platform, set KBS_TRUSTEE_REF, and rerun this snippet." >&2
    exit 1
  fi

  KBS_TRUSTEE_REF="$(
    awk '
      $1 == "coco-trustee:" { in_trustee = 1; next }
      in_trustee && $1 == "version:" {
        gsub(/"/, "", $2)
        print $2
        exit
      }
    ' "${KATA_VERSIONS_FILE}"
  )"
fi

if test -z "${KBS_TRUSTEE_REF}"; then
  echo "ERROR: could not find the coco-trustee version in ${KATA_VERSIONS_FILE}." >&2
  echo "Look up the validated Trustee reference for your platform, set KBS_TRUSTEE_REF, and rerun this snippet." >&2
  exit 1
fi

KBS_COMPOSE_IMAGE_TAG="${KBS_COMPOSE_IMAGE_TAG:-${KBS_TRUSTEE_REF}-${KBS_CLIENT_ARCH}}"
KBS_CLIENT_ARTIFACT="${KBS_CLIENT_IMAGE_REPO}:sample_only-${KBS_COMPOSE_IMAGE_TAG}"
KBS_TRUSTEE_TARBALL_URL="${KBS_TRUSTEE_TARBALL_URL:-https://github.com/confidential-containers/trustee/archive/${KBS_TRUSTEE_REF}.tar.gz}"

skopeo inspect "docker://${KBS_IMAGE_REPO}:${KBS_COMPOSE_IMAGE_TAG}" >/dev/null
skopeo inspect "docker://${KBS_AS_IMAGE_REPO}:${KBS_COMPOSE_IMAGE_TAG}" >/dev/null
skopeo inspect "docker://${KBS_RVPS_IMAGE_REPO}:${KBS_COMPOSE_IMAGE_TAG}" >/dev/null
oras manifest fetch "${KBS_CLIENT_ARTIFACT}" >/dev/null

printf 'KBS_TRUSTEE_REF=%s\nKBS_IMAGE=%s:%s\nAS_IMAGE=%s:%s\nRVPS_IMAGE=%s:%s\nKBS_CLIENT_ARTIFACT=%s\nKBS_URL=%s\n' \
  "${KBS_TRUSTEE_REF}" \
  "${KBS_IMAGE_REPO}" \
  "${KBS_COMPOSE_IMAGE_TAG}" \
  "${KBS_AS_IMAGE_REPO}" \
  "${KBS_COMPOSE_IMAGE_TAG}" \
  "${KBS_RVPS_IMAGE_REPO}" \
  "${KBS_COMPOSE_IMAGE_TAG}" \
  "${KBS_CLIENT_ARTIFACT}" \
  "${KBS_URL}"

If you set KBS_TRUSTEE_REF manually to a release tag or another source reference, also set KBS_COMPOSE_IMAGE_TAG when your Compose images use a different tag naming scheme. The default assumption above matches the staged images referenced by the Kata release metadata. If the matching source tree is not available through the default GitHub archive URL, set KBS_TRUSTEE_TARBALL_URL as well.

Prepare the Trustee Compose Deployment

Download the matching Trustee source tarball to use the Docker Compose logic from the selected Trustee version. The tutorial uses the source tree for the Compose file and the Compose configuration files under kbs/config:

rm -rf "${KBS_TRUSTEE_DIR}"
mkdir -p "${KBS_TRUSTEE_DIR}"

curl -fsSL "${KBS_TRUSTEE_TARBALL_URL}" |
  tar -xz --strip-components=1 -C "${KBS_TRUSTEE_DIR}"

The Compose input files used by this checkpoint are:

• docker-compose.yml, which starts KBS, AS, RVPS, and a one-time setup service that generates demo admin and AS token-signing keys.
• kbs/config/docker-compose/kbs-config.toml, which configures the KBS endpoint, admin authentication, and resource storage.
• kbs/config/as-config.json, which configures AS, including how AS reaches RVPS and the NVIDIA verifier.
• kbs/config/rvps.json, which configures RVPS.

The full files are intentionally not reproduced here. Review the selected source tree, or the same paths in the Trustee repository, if you want to inspect every default.

The upstream Compose file uses latest image tags. Pin the KBS, AS, and RVPS images to the Trustee reference selected above, and publish KBS on the host port selected by KBS_PORT:

sed -i \
  -e "s#image: ghcr.io/confidential-containers/staged-images/kbs-grpc-as:latest#image: ${KBS_IMAGE_REPO}:${KBS_COMPOSE_IMAGE_TAG}#" \
  -e "s#image: ghcr.io/confidential-containers/staged-images/coco-as-grpc:latest#image: ${KBS_AS_IMAGE_REPO}:${KBS_COMPOSE_IMAGE_TAG}#" \
  -e "s#image: ghcr.io/confidential-containers/staged-images/rvps:latest#image: ${KBS_RVPS_IMAGE_REPO}:${KBS_COMPOSE_IMAGE_TAG}#" \
  -e "s#\"8080:8080\"#\"${KBS_PORT}:8080\"#" \
  "${KBS_TRUSTEE_DIR}/docker-compose.yml"

The Compose setup service generates the demo admin keypair and AS token-signing material when the services start. It does not create the KBS endpoint certificate, so this checkpoint generates demo kbs-https.* material and wires it into the KBS configuration. The subject alternative name must match the node IP in KBS_URL so that the guest and kbs-client can verify the HTTPS endpoint:

openssl req -x509 \
  -out "${KBS_CONFIG_DIR}/kbs-https.crt" \
  -keyout "${KBS_CONFIG_DIR}/kbs-https.key" \
  -newkey rsa:2048 \
  -nodes \
  -sha256 \
  -days 365 \
  -subj "/CN=${KBS_NODE_IP}" \
  -addext "subjectAltName=IP:${KBS_NODE_IP},IP:127.0.0.1,DNS:localhost" \
  -addext "basicConstraints=CA:FALSE"

sed -i \
  -e 's#insecure_http = true#insecure_http = false\
private_key = "/opt/confidential-containers/kbs/user-keys/kbs-https.key"\
certificate = "/opt/confidential-containers/kbs/user-keys/kbs-https.crt"#' \
  "${KBS_COMPOSE_CONFIG_DIR}/kbs-config.toml"

grep -q 'insecure_http = false' "${KBS_COMPOSE_CONFIG_DIR}/kbs-config.toml"
grep -q 'type = "Simple"' "${KBS_COMPOSE_CONFIG_DIR}/kbs-config.toml"
jq -e '.verifier_config.nvidia_verifier.type == "Remote"' \
  "${KBS_CONFIG_DIR}/as-config.json" >/dev/null

The demo uses three sets of identity material:

• kbs-https.key and kbs-https.crt identify the KBS HTTPS endpoint.
• private.key and public.pub authenticate admin API calls from kbs-client. KBS is configured with the public key, and kbs-client signs admin requests with the private key. The Compose setup service creates this demo keypair.
• ca.key, ca-cert.pem, token.key, token-cert.pem, and token-cert-chain.pem let AS sign attestation tokens that KBS can verify. The Compose setup service creates this demo token-signing material. In this single-node showcase, KBS and AS are started together, so this material is not used to express an independent operational boundary. It becomes important in more advanced deployments, for example when KBS and AS are operated separately and KBS must trust tokens issued by that AS.

For production, decide who owns each of these identities. For example, the KBS endpoint certificate should usually come from an organizational CA, admin signing keys should be issued and protected by an approved identity process, and token-signing private keys may need to be generated or protected by an HSM or another managed key service.

Deploy Trustee

Start the Compose deployment. The --project-directory option makes the relative volume paths in docker-compose.yml resolve inside the downloaded Trustee tree. The Compose file also contains build: sections for local development, so the commands below pull the pinned images first and use --no-build to keep this tutorial on pre-built artifacts:

docker compose \
  --project-directory "${KBS_TRUSTEE_DIR}" \
  --project-name nim-trustee \
  pull kbs as rvps setup

docker compose \
  --project-directory "${KBS_TRUSTEE_DIR}" \
  --project-name nim-trustee \
  up -d --no-build

Verify the Compose services. This Trustee version does not expose a dedicated health endpoint, so the functional KBS check is done after kbs-client is installed:

docker compose \
  --project-directory "${KBS_TRUSTEE_DIR}" \
  --project-name nim-trustee \
  ps

Keep KBS_URL, the admin private key, and the KBS HTTPS certificate; later checkpoints use them to configure KBS resources and to build pod initdata:

KBS_ADMIN_PRIVATE_KEY_FILE="${KBS_CONFIG_DIR}/private.key"
KBS_CERT_FILE="${KBS_CONFIG_DIR}/kbs-https.crt"
test -s "${KBS_ADMIN_PRIVATE_KEY_FILE}"
test -s "${KBS_CERT_FILE}"

Install kbs-client

Install kbs-client for KBS admin operations. The variables above set KBS_CLIENT_ARTIFACT to the staged kbs-client artifact built from the selected Trustee reference. The oras pull command below downloads that artifact and extracts the kbs-client binary into ${KBS_WORKDIR}/bin. This keeps the client matched to the deployed KBS service, including the admin authentication mode. This does not change KBS state yet; the actual resource provisioning happens in the next checkpoint.

kbs-client is an admin tool used to provision and update KBS resources, reference values, and resource policy. This tutorial runs the extracted binary from the worker host; production deployments should run equivalent admin operations from a trusted operator environment.

command -v oras >/dev/null

mkdir -p "${KBS_WORKDIR}/bin"

oras pull \
  --output "${KBS_WORKDIR}/bin" \
  "${KBS_CLIENT_ARTIFACT}"

chmod +x "${KBS_WORKDIR}/bin/kbs-client"
KBS_CLIENT="${KBS_WORKDIR}/bin/kbs-client"

"${KBS_CLIENT}" --version

Verify the HTTPS endpoint and admin authentication by writing and then deleting a non-secret probe resource:

printf 'checkpoint2-probe\n' > "${KBS_WORKDIR}/checkpoint2-probe.txt"

"${KBS_CLIENT}" \
  --cert-file "${KBS_CERT_FILE}" \
  --url "${KBS_URL}" \
  config \
  --auth-private-key "${KBS_ADMIN_PRIVATE_KEY_FILE}" \
  set-resource \
  --path default/checkpoint2/probe \
  --resource-file "${KBS_WORKDIR}/checkpoint2-probe.txt"

"${KBS_CLIENT}" \
  --cert-file "${KBS_CERT_FILE}" \
  --url "${KBS_URL}" \
  config \
  --auth-private-key "${KBS_ADMIN_PRIVATE_KEY_FILE}" \
  delete-resource \
  --path default/checkpoint2/probe

Keep this shell, or save KBS_CLIENT, KBS_WORKDIR, KBS_ADMIN_PRIVATE_KEY_FILE, KBS_CERT_FILE, and KBS_URL somewhere private so they can be reused while working through the next checkpoints. Do not remove KBS_WORKDIR; it contains the Trustee source tree, Compose configuration, demo state, and generated tutorial files. Later kbs-client commands authenticate to KBS with the Compose-generated admin private key, so protect it as an administrator credential.

Checkpoint 3: Prepare the TEE NIM Manifest and Sealed Secret

This checkpoint creates the signed sealed secret value for the runtime NGC API key, then renders the TEE NIM Pod manifest without deploying it yet. The manifest will use the SNP GPU runtime class, a Kubernetes image-pull Secret reference, a Kubernetes Secret that carries the sealed runtime secret, the trusted storage PVC, and the probes from the baseline manifest. The image-pull Secret and PVC are created later, before the Pod is deployed.

Generate the manifest before deployment so genpolicy can use the Pod specification as input. The generated Kata agent policy and generated cc_init_data depend on this manifest, and the measurement checkpoint will reuse that generated cc_init_data.

This workload is configured with trusted image storage. With guest pull, the guest needs a place for large NIM model data and container image layers. Keeping that data in guest memory-backed filesystems would require a much larger confidential VM. Putting it on a block device attached to the confidential VM gives the guest a storage target it can own and use for large artifacts without relying on the host’s normal container filesystem as the runtime storage boundary.

There are two NGC-related Secrets:

• ngc-secret-instruct is an image-pull Secret used by the Kubernetes and CRI layer for authenticated registry metadata access. In production, minimize the scope of this token and consider whether your registry setup can avoid exposing broad credentials to the workload cluster control plane. It is created later.
• ngc-api-key-sealed-instruct contains only a signed sealed value. Kubernetes sees the sealed value; CDH unseals it inside the guest by fetching kbs:///default/ngc-api-key/instruct from KBS. This Secret is included in the manifest rendered below because genpolicy resolves secretKeyRef values from Secret objects in its YAML input.

Generate a P-256 signing key for sealed secrets. The private JWK stays outside KBS; only the public JWK is uploaded later. CDH uses the public key resource to verify that the sealed NIM runtime secret came from the expected operator key before resolving the referenced plaintext secret from KBS. In production, generate and sign sealed runtime secrets in a trusted release pipeline, protect the private signing key, and publish only the sealed value through the approved workload release path.

KBS_WORKDIR="${KBS_WORKDIR}" python3 - <<'EOF'
import os
from jwcrypto import jwk

workdir = os.environ["KBS_WORKDIR"]
k = jwk.JWK.generate(
    kty='EC', crv='P-256', alg='ES256',
    use='sig', kid='sealed-secret-nim-key')
open(f'{workdir}/signing-key-private.jwk', 'w').write(k.export_private())
open(f'{workdir}/signing-key-public.jwk', 'w').write(k.export_public())
EOF

Create a signed vault sealed secret that points to the KBS-hosted NIM runtime API key. Under normal circumstances, use the secret CLI from the Confidential Containers project to create this value. Because this document is intended to avoid requiring readers to build CoCo components from scratch, and because the secret CLI is not published as a standalone artifact at the time of writing, the next snippet is a temporary compatibility helper that emits the same vault sealed-secret format. Do not use this helper as production secret tooling; use the supported CoCo tooling from a trusted release pipeline instead. A vault sealed secret is a JWS-signed JSON document containing the KBS resource URI and provider metadata. The protected header includes b64 to match the format produced by the secret CLI tool.

KBS_WORKDIR="${KBS_WORKDIR}" python3 - <<'EOF'
import base64
import json
import os
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.asymmetric import ec, utils

workdir = os.environ["KBS_WORKDIR"]
signing_jwk = json.loads(open(f"{workdir}/signing-key-private.jwk").read())

def b64url_decode(value):
    return base64.urlsafe_b64decode(value + "=" * (-len(value) % 4))

def b64url_encode(value):
    return base64.urlsafe_b64encode(value).rstrip(b"=").decode()

payload = {
    "version": "0.1.0",
    "type": "vault",
    "name": "kbs:///default/ngc-api-key/instruct",
    "provider": "kbs",
    "provider_settings": {},
    "annotations": {},
}

protected = {
    "b64": True,
    "alg": "ES256",
    "kid": "kbs:///default/signing-key/sealed-secret",
}

private_value = int.from_bytes(b64url_decode(signing_jwk["d"]), "big")
private_key = ec.derive_private_key(private_value, ec.SECP256R1())

protected_b64 = b64url_encode(json.dumps(
    protected, separators=(",", ":")).encode())
payload_b64 = b64url_encode(json.dumps(payload, separators=(",", ":")).encode())
signing_input = f"{protected_b64}.{payload_b64}".encode()
signature_der = private_key.sign(signing_input, ec.ECDSA(hashes.SHA256()))
r, s = utils.decode_dss_signature(signature_der)
signature = r.to_bytes(32, "big") + s.to_bytes(32, "big")

with open(f"{workdir}/ngc-api-key-instruct.sealed", "w") as f:
    f.write(f"sealed.{protected_b64}.{payload_b64}.{b64url_encode(signature)}")
EOF

Render the Pod manifest:

NIM_TEE_MANIFEST="${KBS_WORKDIR}/nvidia-nim-llama-3-1-8b-instruct-tee.yaml"

SEALED_NGC_API_KEY_BASE64="$(
  base64 -w0 "${KBS_WORKDIR}/ngc-api-key-instruct.sealed"
)"

cat > "${NIM_TEE_MANIFEST}" <<EOF
apiVersion: v1
kind: Secret
metadata:
  name: ngc-api-key-sealed-instruct
type: Opaque
data:
  api-key: "${SEALED_NGC_API_KEY_BASE64}"
---
apiVersion: v1
kind: Pod
metadata:
  name: nvidia-nim-llama-3-1-8b-instruct
  labels:
    app: nvidia-nim-llama-3-1-8b-instruct
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 1000
    fsGroup: 1000
    supplementalGroups: [4, 20, 24, 25, 27, 29, 30, 44, 46]
  restartPolicy: Never
  runtimeClassName: kata-qemu-nvidia-gpu-snp
  imagePullSecrets:
    - name: ngc-secret-instruct
  containers:
    - name: nvidia-nim-llama-3-1-8b-instruct
      image: nvcr.io/nim/meta/llama-3.1-8b-instruct:1.13.1
      ports:
        - containerPort: 8000
          name: http-openai
      livenessProbe:
        httpGet:
          path: /v1/health/live
          port: http-openai
        initialDelaySeconds: 15
        periodSeconds: 10
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 3
      readinessProbe:
        httpGet:
          path: /v1/health/ready
          port: http-openai
        initialDelaySeconds: 15
        periodSeconds: 10
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 3
      startupProbe:
        httpGet:
          path: /v1/health/ready
          port: http-openai
        initialDelaySeconds: 360
        periodSeconds: 10
        timeoutSeconds: 1
        successThreshold: 1
        failureThreshold: 120
      env:
        - name: NGC_API_KEY
          valueFrom:
            secretKeyRef:
              name: ngc-api-key-sealed-instruct
              key: api-key
      resources:
        limits:
          nvidia.com/pgpu: "1"
          cpu: "16"
          memory: "48Gi"
      volumeMounts:
        - name: nim-trusted-cache
          mountPath: /opt/nim/.cache
      volumeDevices:
        - name: trusted-storage
          devicePath: /dev/trusted_store
  volumes:
    - name: nim-trusted-cache
      emptyDir:
        sizeLimit: 64Gi
    - name: trusted-storage
      persistentVolumeClaim:
        claimName: trusted-pvc-instruct
EOF

Validate the manifest without creating anything:

kubectl apply --dry-run=server -f "${NIM_TEE_MANIFEST}"

The server dry-run should report the Secret and Pod as created or configured in dry-run mode. The referenced image-pull Secret and trusted storage PVC do not need to exist yet for this validation; they are created before the workload is deployed.

The cc_init_data annotation is still absent at this point. Checkpoint 5 adds it by running genpolicy with nim-initdata.toml.

Checkpoint 4: Create the Pod Initdata Annotation

This checkpoint will add the AA and CDH configuration used by the guest. The initdata will include the KBS address, the KBS certificate, the registry credential URI for authenticated registry access, and the image signature policy URI.

The host still supplies this initdata through a Pod annotation, so the guest must not trust it merely because it was delivered. For SNP launches, Kata hashes the generated initdata document and configures QEMU with that digest as the SNP HOST_DATA launch value. SNP reports this value in attestation evidence. The guest verifies that the initdata it received hashes to the reported HOST_DATA value, and the attestation flow exposes the verified digest to KBS. Checkpoint 7 pins that digest in the KBS resource policy.

The KBS URL must be the same address used in checkpoint 2, and the certificate must be the certificate that KBS serves for that address. The certificate is included in both aa.toml and cdh.toml because AA and CDH are separate guest components that each connect to the HTTPS KBS endpoint. AA uses its KBS configuration for attestation, while CDH uses its KBS configuration to fetch resources and unseal signed sealed secrets.

Create nim-initdata.toml:

INITDATA_FILE="${KBS_WORKDIR}/nim-initdata.toml"

{
  cat <<EOF
version = "0.1.0"
algorithm = "sha256"

[data]
"aa.toml" = """
[token_configs]
[token_configs.kbs]
url = "${KBS_URL}"
cert = '''
EOF
  cat "${KBS_CERT_FILE}"
  cat <<EOF
'''
"""

"cdh.toml" = """
[kbc]
name = "cc_kbc"
url = "${KBS_URL}"
kbs_cert = '''
EOF
  cat "${KBS_CERT_FILE}"
  cat <<'EOF'
'''

[image]
max_concurrent_layer_downloads_per_image = 1
authenticated_registry_credentials_uri = "kbs:///default/credentials/nvcr"
image_security_policy_uri = "kbs:///default/security-policy/nim"
"""
EOF
} > "${INITDATA_FILE}"

Do not add this file directly to the Pod yet. Checkpoint 5 uses it as input to genpolicy; genpolicy will add the Kata agent policy to the same initdata document and then write the generated io.katacontainers.config.hypervisor.cc_init_data annotation.

Checkpoint 5: Generate the Kata Agent Policy

This checkpoint will run genpolicy against the TEE NIM Pod manifest and attach the generated Kata agent policy to the Pod initdata. The environment running genpolicy needs registry access to nvcr.io, because genpolicy reads image metadata while constructing the policy.

Treat genpolicy as part of the trusted build or release process. Its output polices the shim-to-agent interface at the trust boundary, so production deployments should generate, review, and publish this policy from a trusted environment rather than interactively on a worker node.

genpolicy reads registry credentials through Docker-style credential lookup. The example below creates a temporary Docker config for nvcr.io in the genpolicy work directory and points genpolicy at it. If the trusted build environment already has suitable Docker credentials, you can use those instead.

Fetch the official Kata tools release asset that matches the installed Kata runtime.

GENPOLICY_WORKDIR="${KBS_WORKDIR}/genpolicy"
NIM_TEE_MANIFEST="${NIM_TEE_MANIFEST:-${KBS_WORKDIR}/nvidia-nim-llama-3-1-8b-instruct-tee.yaml}"
INITDATA_FILE="${INITDATA_FILE:-${KBS_WORKDIR}/nim-initdata.toml}"
NIM_POLICY_MANIFEST="${GENPOLICY_WORKDIR}/nvidia-nim-llama-3-1-8b-instruct-tee-policy.yaml"
GENPOLICY_INITDATA="${GENPOLICY_WORKDIR}/nim-initdata.toml"
GENPOLICY_DOCKER_CONFIG="${GENPOLICY_WORKDIR}/docker-config"
KATA_RUNTIME_VERSION="$(/opt/kata/bin/kata-runtime --version \
  | awk '/kata-runtime/ {print $3}')"
KATA_TOOLS_ARCH="amd64"
KATA_TOOLS_NAME="kata-tools-static-${KATA_RUNTIME_VERSION}-${KATA_TOOLS_ARCH}"
KATA_TOOLS_TARBALL="${KATA_TOOLS_NAME}.tar.zst"
KATA_RELEASE_BASE="https://github.com/kata-containers/kata-containers/releases/download"
KATA_TOOLS_URL="${KATA_RELEASE_BASE}/${KATA_RUNTIME_VERSION}/${KATA_TOOLS_TARBALL}"
KATA_TOOLS_EXTRACT_DIR="${GENPOLICY_WORKDIR}/kata-tools"
KATA_TOOLS_DIR="${KATA_TOOLS_EXTRACT_DIR}/opt/kata"
GENPOLICY_BIN="${KATA_TOOLS_DIR}/bin/genpolicy"
GENPOLICY_DEFAULTS="${KATA_TOOLS_DIR}/share/defaults/kata-containers"

mkdir -p "${GENPOLICY_WORKDIR}/genpolicy-settings.d" \
  "${GENPOLICY_DOCKER_CONFIG}" \
  "${KATA_TOOLS_EXTRACT_DIR}"

curl -fL \
  -o "${GENPOLICY_WORKDIR}/${KATA_TOOLS_TARBALL}" \
  "${KATA_TOOLS_URL}"

tar --zstd \
  -xf "${GENPOLICY_WORKDIR}/${KATA_TOOLS_TARBALL}" \
  -C "${KATA_TOOLS_EXTRACT_DIR}"

"${GENPOLICY_BIN}" -v \
  -j "${GENPOLICY_DEFAULTS}"

Prepare registry credentials and copy the release-matched genpolicy defaults into the local work directory:

: "${NGC_API_KEY:?set NGC_API_KEY to an NGC API key that can pull the NIM image}"
export NGC_API_KEY

GENPOLICY_AUTH="$(printf '$oauthtoken:%s' "${NGC_API_KEY}" | base64 -w0)"
jq -n \
  --arg username '$oauthtoken' \
  --arg password "${NGC_API_KEY}" \
  --arg auth "${GENPOLICY_AUTH}" \
  '{auths: {"nvcr.io": {username: $username, password: $password, auth: $auth}}}' \
  > "${GENPOLICY_DOCKER_CONFIG}/config.json"

cp "${GENPOLICY_DEFAULTS}/rules.rego" \
  "${GENPOLICY_WORKDIR}/rules.rego"

cp "${GENPOLICY_DEFAULTS}/genpolicy-settings.json" \
  "${GENPOLICY_WORKDIR}/genpolicy-settings.json"

cp "${GENPOLICY_DEFAULTS}/drop-in-examples/20-oci-1.3.0-drop-in.json" \
  "${GENPOLICY_WORKDIR}/genpolicy-settings.d/20-oci-1.3.0-drop-in.json"

cp "${NIM_TEE_MANIFEST}" "${NIM_POLICY_MANIFEST}"
cp "${INITDATA_FILE}" "${GENPOLICY_INITDATA}"

The OCI 1.3.0 drop-in matches the GPU/SNP CI setup. It adjusts the expected OCI version in the generated policy. If your containerd stack uses a different OCI version, choose the matching drop-in or settings value.

Optional: allow kubectl logs during the workload run by enabling ReadStreamRequest in the generated Kata agent policy. This is useful while debugging the showcase, but production policy should usually leave host-side log streaming disabled unless there is an explicit operational requirement.

cat > "${GENPOLICY_WORKDIR}/genpolicy-settings.d/99-observation-read-stream.json" <<'EOF'
[
  {
    "op": "replace",
    "path": "/request_defaults/ReadStreamRequest",
    "value": true
  }
]
EOF

Run genpolicy. It updates NIM_POLICY_MANIFEST in place by adding io.katacontainers.config.hypervisor.cc_init_data; that annotation contains the initdata from checkpoint 4 plus the generated policy.rego.

(
  cd "${GENPOLICY_WORKDIR}"

  DOCKER_CONFIG="${GENPOLICY_DOCKER_CONFIG}" \
  RUST_LOG=info \
  "${GENPOLICY_BIN}" \
    -u \
    -y "${NIM_POLICY_MANIFEST}" \
    -p "${GENPOLICY_WORKDIR}/rules.rego" \
    -j "${GENPOLICY_WORKDIR}" \
    --initdata-path="${GENPOLICY_INITDATA}"
)

Validate the generated manifest without creating anything:

kubectl apply --dry-run=server -f "${NIM_POLICY_MANIFEST}"

Inspect the generated cc_init_data annotation:

kubectl create --dry-run=client \
  -f "${NIM_POLICY_MANIFEST}" \
  -o json \
  | jq -r '
      select(.kind == "Pod")
      | .metadata.annotations["io.katacontainers.config.hypervisor.cc_init_data"]
    ' \
  | base64 -d \
  | gzip -d > "${GENPOLICY_WORKDIR}/generated-initdata.toml"

grep -n '^"aa\.toml"[[:space:]]*=' "${GENPOLICY_WORKDIR}/generated-initdata.toml"
grep -n '^"cdh\.toml"[[:space:]]*=' "${GENPOLICY_WORKDIR}/generated-initdata.toml"
grep -n '^"policy\.rego"[[:space:]]*=' "${GENPOLICY_WORKDIR}/generated-initdata.toml"

The generated initdata must contain aa.toml, cdh.toml, and policy.rego. Keep NIM_POLICY_MANIFEST; the measurement checkpoint reuses its generated cc_init_data, and the deployment checkpoint deploys it as the TEE NIM workload.

After this checkpoint, keep the fully generated manifest available, but do not deploy the NIM workload until the reference values, KBS resources, KBS policy, and trusted storage resources have been prepared.

Checkpoint 6: Collect SNP Launch Reference Values

This checkpoint collects the SNP launch inputs needed by the KBS policy. It is only needed in this manual demo when the launch reference values are not already approved and available. In production, these values should be computed and approved ahead of time from trusted release artifacts and the expected VM launch configuration, so the KBS policy can be installed before the workload is ever deployed.

If you already have approved values, provide them in ${KBS_WORKDIR}/nim-snp-reference-values.env using the variable names written below and continue with checkpoint 7 instead of launching the measurement Pod.

The collection Pod can be lightweight: it does not need to run the NIM image or become application-ready. It only needs to start the same kind of confidential VM so the launch inputs can be recorded. Keep the launch-relevant settings aligned with the TEE NIM Pod, especially the runtime class, generated cc_init_data annotation, GPU request, and CPU/memory sizing, because those can affect the QEMU command line and therefore the SNP launch measurement. The container image can be a small sleep image.

This checkpoint derives the expected launch measurement from the VM that Kata starts for the collection Pod. The later steps locate the measurement Pod’s QEMU process, read the SEV-SNP launch inputs from that process, and pass them to sev-snp-measure. This ties the recorded reference value to the VM shape created by Kubernetes and Kata for this workload.

The generated cc_init_data matters. Kata hashes that document and configures QEMU with the digest as the SNP HOST_DATA launch value; the KBS resource policy will pin that digest.

First, confirm the node is in confidential GPU mode and extract the generated cc_init_data from the policy-bearing manifest:

kubectl get nodes \
  -L nvidia.com/cc.mode,nvidia.com/cc.mode.state,nvidia.com/cc.ready.state

MEASUREMENT_POD_NAME="nim-snp-measurement"
MEASUREMENT_POD_MANIFEST="${KBS_WORKDIR}/nim-snp-measurement-pod.yaml"
NODE_NAME="$(
  kubectl get nodes \
    -l nvidia.com/cc.mode=on,nvidia.com/cc.mode.state=on,nvidia.com/cc.ready.state=true \
    -o jsonpath='{.items[0].metadata.name}'
)"

CC_INIT_DATA="$(
  kubectl create --dry-run=client \
    -f "${NIM_POLICY_MANIFEST}" \
    -o json \
    | jq -r '
        select(.kind == "Pod")
        | .metadata.annotations["io.katacontainers.config.hypervisor.cc_init_data"]
      '
)"

test -n "${NODE_NAME}"
test -n "${CC_INIT_DATA}"
test "${CC_INIT_DATA}" != "null"

Create a lightweight measurement Pod. The limits mirror the TEE NIM Pod’s CPU, memory, and GPU requests, but the image is a small sleep container:

cat > "${MEASUREMENT_POD_MANIFEST}" <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: ${MEASUREMENT_POD_NAME}
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: "${CC_INIT_DATA}"
spec:
  restartPolicy: Never
  runtimeClassName: kata-qemu-nvidia-gpu-snp
  nodeName: ${NODE_NAME}
  containers:
    - name: sleep
      image: quay.io/prometheus/busybox:latest
      imagePullPolicy: IfNotPresent
      command: ["sh", "-c", "sleep 600"]
      resources:
        limits:
          nvidia.com/pgpu: "1"
          cpu: "16"
          memory: "48Gi"
EOF

kubectl delete pod "${MEASUREMENT_POD_NAME}" --ignore-not-found
kubectl apply -f "${MEASUREMENT_POD_MANIFEST}"

This checkpoint uses two host-side tools that are not installed by Kata itself: sev-snp-measure computes the expected SNP launch measurement from the QEMU launch inputs, and snphost reads the host’s reported SNP TCB values. Install them from your distribution or approved CI image if available. Otherwise, one common developer setup is shown below. Some distributions block pip install --user for externally managed Python installations; in that case, use a virtual environment or a distribution package instead of overriding the system policy. The snphost installation command uses crates.io and requires cargo from a Rust toolchain. cargo install writes the snphost binary to ${HOME}/.cargo/bin, so add that directory to PATH for the commands that follow.