Google Cloud Platform (GCP)

This documentation will walk you through setting up Cloud API Adaptor (CAA) (a.k.a. Peer Pods) on Google Kubernetes Engine (GKE).

It explains how to deploy:

• A single worker node Kubernetes cluster using Google Kubernetes Engine (GKE),
• CAA on that Kubernetes cluster,
• A sample application deployed using CAA to verify that everything is working as expected.

Before you begin

Complete the common prerequisites, including the Google Cloud CLI.

Google Cloud Project:

• Ensure you have a Google Cloud project created,
• Note the Project ID (export it as GCP_PROJECT_ID).

GKE Preparation

1. Set the environment variable GCP_PROJECT_ID to your Google Cloud project ID:

   export GCP_PROJECT_ID="YOUR_PROJECT_ID"
   

2. Authenticate with Google Cloud and set the project:

   gcloud auth login
   gcloud config set project "${GCP_PROJECT_ID}"
   

3. Enable the GKE API:

   gcloud services enable container.googleapis.com \
     --project="${GCP_PROJECT_ID}"
   

   This API is required to create and manage the GKE cluster.

4. Set the region and cluster name:

   export GCP_REGION="us-central1"
   export CLUSTER_NAME="caa-gke"
   

   Note

   “us-central1” was chosen because supports Confidential VMs.
   For a complete list of supported regions visit supported-configurations.

Setup cluster

Deploy a single node Kubernetes cluster using GKE:

gcloud container clusters create "${CLUSTER_NAME}" \
  --zone ${GCP_REGION}-a \
  --machine-type "e2-standard-4" \
  --image-type UBUNTU_CONTAINERD \
  --num-nodes 1

Note: The e2-standard-4 machine type is used for the GKE cluster nodes, which is a general-purpose machine.
The UBUNTU_CONTAINERD image type is specified to ensure compatibility with the container runtime used by CAA.

Get cluster credentials:

gcloud container clusters get-credentials "${CLUSTER_NAME}" \
  --zone "${GCP_REGION}-a" \
  --project "${GCP_PROJECT_ID}"

(Optional) Verify that the cluster is reachable:

kubectl get nodes -o wide

Label the worker node:

kubectl get nodes \
  --selector='!node-role.kubernetes.io/master' \
  -o name \
  | xargs -I{} kubectl label {} node.kubernetes.io/worker=

This labeling step adds the node.kubernetes.io/worker label to non-control-plane nodes.

Configure VPC network

We need to make sure port 15150 is open under the default VPC network:

gcloud compute firewall-rules create allow-port-15150 \
    --project=${GCP_PROJECT_ID} \
    --network=default \
    --allow=tcp:15150

For production scenarios, it is advisable to restrict the source IP range to minimize security risks. For example, you can restrict the source range to a specific IP address or CIDR block:

gcloud compute firewall-rules create allow-port-15150-restricted \
   --project=${GCP_PROJECT_ID} \
   --network=default \
   --allow=tcp:15150 \
   --source-ranges=[YOUR_EXTERNAL_IP]

Create the PodVM image

This section describes how to create the PodVM image for your workload and publish it on Google Cloud Storage.

Build the PodVM image

• Build methods:
• Release
• Debug
• Pre-built

Publish image to Google Storage

1. Export the bucket name:

   export BUCKET_NAME="peerpods-bucket"
   

   Note: GCP_PROJECT_ID and GCP_REGION should already be set from the GCP Preparation section.
   The BUCKET_NAME should be globally unique across all of Google Cloud, so consider adding a random suffix if needed.

2. Create a GCS bucket for images:

   gcloud storage buckets create "gs://${BUCKET_NAME}" \
     --project="${GCP_PROJECT_ID}" \
     --location="${GCP_REGION}"
   

3. Upload the disk image to a bucket and create the image:

   1. Prepare image name:

      export IMAGE_BASE_NAME="podvm-image"
      export CAA_HASH=$(git rev-parse --short HEAD 2>/dev/null || echo "")
      export IMAGE_NAME="${IMAGE_BASE_NAME}-${CAA_HASH:-$(date +%s)}-release"
      

      Note: The image name uses the git commit hash if available (for Release/Debug builds), or falls back to a Unix timestamp (for Pre-built images).

   2. Upload image to GCS bucket:

      gcloud storage cp build/disk.tar.gz gs://${BUCKET_NAME}/peerpods-disk.tar.gz
      

   3. Create image in GCP with defined name from the uploaded disk image:

      • AMD SEV-SNP
      • Intel® TDX

      gcloud compute images create ${IMAGE_NAME} \
        --source-uri=gs://${BUCKET_NAME}/peerpods-disk.tar.gz \
        --guest-os-features=UEFI_COMPATIBLE
      

      This command creates a new image in GCP with the specified name and the uploaded disk image.
      The --guest-os-features flag ensures that the image is compatible with UEFI.

      gcloud compute images create ${IMAGE_NAME} \
        --source-uri=gs://${BUCKET_NAME}/peerpods-disk.tar.gz \
        --guest-os-features=UEFI_COMPATIBLE,TDX_CAPABLE
      

      Both UEFI_COMPATIBLE and TDX_CAPABLE are required for tdx TEE.

      This command creates a new image in GCP with the specified name and the uploaded disk image.
      The --guest-os-features flag ensures that the image is compatible with UEFI and TDX.

Export PodVM image id

Export the PodVM image id to be used in the provider configuration. This is the name of the image created in the previous step.

export PODVM_IMAGE_ID="$IMAGE_NAME"

gcloud compute images list \
  --project="${GCP_PROJECT_ID}" \
  --filter="name ~ ^${IMAGE_BASE_NAME}-" \
  --sort-by=~creationTimestamp \
  --limit=1 \
  --format="value(name)"

Deploy

Preparation

Complete Prepare the CAA deployment.

This downloads the Helm artifacts, selects the CAA image, installs cert-manager, and creates the installation namespace.

Note: Keep the resulting CAA_VERSION, CAA_IMAGE, and CAA_TAG values in your terminal, then return here.

Set TEE Platform Configuration

Set TEE platform and PodVM instance type for your workload:

• AMD SEV-SNP
• Intel® TDX
• Non-Confidential

export PODVM_INSTANCE_TYPE="n2d-standard-4"
export DISABLECVM=false
export GCP_CONFIDENTIAL_TYPE="SEV" # SEV or SEV_SNP
export GCP_DISK_TYPE="pd-standard"

export PODVM_INSTANCE_TYPE="c3-standard-4"
export DISABLECVM=false
export GCP_CONFIDENTIAL_TYPE="TDX"
export GCP_DISK_TYPE="pd-balanced"

export PODVM_INSTANCE_TYPE="e2-medium"
export DISABLECVM=true
export GCP_CONFIDENTIAL_TYPE=""
export GCP_DISK_TYPE="pd-standard"

Populate the provider file

List of all available configuration options can be found in two places:

• Main charts values
• GCP specific values

Run the following command to update the providers/gcp.yaml file:

cat <<EOF > providers/gcp.yaml
provider: gcp
image:
  name: "${CAA_IMAGE}"
  tag: "${CAA_TAG}"
providerConfigs:
  gcp:
    GCP_NETWORK: "global/networks/default"
    GCP_PROJECT_ID: "${GCP_PROJECT_ID}"
    GCP_ZONE: "${GCP_REGION}-a"
    GCP_MACHINE_TYPE: "${PODVM_INSTANCE_TYPE}"
    GCP_DISK_TYPE: "${GCP_DISK_TYPE}"
    PODVM_IMAGE_NAME: "${PODVM_IMAGE_ID}"
    GCP_CONFIDENTIAL_TYPE: "${GCP_CONFIDENTIAL_TYPE}"
    DISABLECVM: ${DISABLECVM}
EOF

Configure Authentication

Choose how CAA authenticates with GCP. You can use either static credentials (service account key file) or Workload Identity Federation (WIF).

Note: Workload Identity Federation (WIF) is the recommended authentication method for GKE deployments. With WIF, the CAA pods authenticate via OIDC — no static GCP credentials are stored in Kubernetes secrets.

Create GCP Service Account

Create a GCP service account with the necessary permissions for CAA to manage peer pod VMs:

export GSA_NAME="cloud-api-adaptor"
export GSA_EMAIL="${GSA_NAME}@${GCP_PROJECT_ID}.iam.gserviceaccount.com"

gcloud iam service-accounts create ${GSA_NAME} \
  --description="Cloud API Adaptor Service Account" \
  --display-name="Cloud API Adaptor Service Account" \
  --project=${GCP_PROJECT_ID}

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/compute.instanceAdmin.v1"

gcloud projects add-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/iam.serviceAccountUser"

These roles allow the Cloud API Adaptor to:

• create, start/stop, and delete the Compute Engine instances used as PodVMs (roles/compute.instanceAdmin.v1)
• run actions as the service account when provisioning those resources (roles/iam.serviceAccountUser)

Note: IAM policy updates can take a few minutes to propagate. If later steps fail with permission errors, wait briefly and retry.

Choose Authentication Method

• Static Credentials
• Workload Identity Federation (GKE)

export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/peerpods_application_key.json

gcloud iam service-accounts keys create \
  ${GOOGLE_APPLICATION_CREDENTIALS} \
  --iam-account=${GSA_EMAIL}

gcloud services enable compute.googleapis.com iam.googleapis.com iamcredentials.googleapis.com sts.googleapis.com \
  --project=${GCP_PROJECT_ID}

export CAA_K8S_SERVICE_ACCOUNT="cloud-api-adaptor"
export CAA_NAMESPACE="confidential-containers-system"
export GKE_WORKLOAD_POOL="${GCP_PROJECT_ID}.svc.id.goog"

gcloud container clusters describe ${CLUSTER_NAME} \
  --zone=${GCP_REGION}-a \
  --format="value(workloadIdentityConfig.workloadPool)"

gcloud container clusters update ${CLUSTER_NAME} \
  --zone=${GCP_REGION}-a \
  --workload-pool=${GKE_WORKLOAD_POOL}

# Note: Use 'locations' instead of 'zones' for the OIDC issuer URL
export OIDC_ISSUER="https://container.googleapis.com/v1/projects/${GCP_PROJECT_ID}/locations/${GCP_REGION}-a/clusters/${CLUSTER_NAME}"

export PROJECT_NUMBER=$(gcloud projects describe ${GCP_PROJECT_ID} \
  --format="value(projectNumber)")

echo "OIDC Issuer: ${OIDC_ISSUER}"
echo "Project Number: ${PROJECT_NUMBER}"

# Verify the OIDC endpoint is accessible
curl -s "${OIDC_ISSUER}/.well-known/openid-configuration" | jq .issuer

export WI_POOL_NAME="caa-direct-wif-pool"

gcloud iam workload-identity-pools create ${WI_POOL_NAME} \
  --location=global \
  --description="Workload Identity pool for cloud-api-adaptor with hostNetwork true" \
  --display-name="CAA Direct WIF Pool" \
  --project=${GCP_PROJECT_ID}

export OIDC_PROVIDER_NAME="${CLUSTER_NAME}-oidc-provider"

gcloud iam workload-identity-pools providers create-oidc ${OIDC_PROVIDER_NAME} \
  --project=${GCP_PROJECT_ID} \
  --location=global \
  --workload-identity-pool=${WI_POOL_NAME} \
  --issuer-uri="${OIDC_ISSUER}" \
  --allowed-audiences="${GKE_WORKLOAD_POOL}" \
  --attribute-mapping="google.subject=assertion.sub,attribute.namespace=assertion['kubernetes.io']['namespace'],attribute.service_account_name=assertion['kubernetes.io']['serviceaccount']['name']" \
  --attribute-condition="assertion.sub.startsWith('system:serviceaccount:')"

gcloud iam service-accounts add-iam-policy-binding ${GSA_EMAIL} \
  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${WI_POOL_NAME}/subject/system:serviceaccount:${CAA_NAMESPACE}:${CAA_K8S_SERVICE_ACCOUNT}" \
  --role="roles/iam.workloadIdentityUser" \
  --project=${GCP_PROJECT_ID}

gcloud iam service-accounts add-iam-policy-binding ${GSA_EMAIL} \
  --member="principal://iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${WI_POOL_NAME}/subject/system:serviceaccount:${CAA_NAMESPACE}:${CAA_K8S_SERVICE_ACCOUNT}" \
  --role="roles/iam.serviceAccountTokenCreator" \
  --project=${GCP_PROJECT_ID}

export PROVIDER_NAME="//iam.googleapis.com/projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${WI_POOL_NAME}/providers/${OIDC_PROVIDER_NAME}"

echo "Provider Name: ${PROVIDER_NAME}"
echo "GSA Email: ${GSA_EMAIL}"

Deploy helm chart

1. Create a ResourceQuota to allow system priority classes:

   GKE enforces strict resource quota policies. The kata-deploy DaemonSet requires system priority classes (system-node-critical or system-cluster-critical) to ensure proper scheduling and execution. Create a ResourceQuota to allow these priority classes:

   kubectl apply -f - << EOF
   apiVersion: v1
   kind: ResourceQuota
   metadata:
     name: allow-system-priority-classes
     namespace: confidential-containers-system
   spec:
     hard:
       pods: "100"
     scopeSelector:
       matchExpressions:
       - operator: In
         scopeName: PriorityClass
         values:
         - system-node-critical
         - system-cluster-critical
   EOF
   

2. Create credentials and install the Helm chart:

   The -f and --set customization options are described in advanced configuration.

• Static Credentials
• Workload Identity Federation (GKE)

kubectl create secret generic my-provider-creds \
  -n confidential-containers-system \
  --from-file=GCP_CREDENTIALS="${GOOGLE_APPLICATION_CREDENTIALS}"

helm install peerpods . \
  -f providers/gcp.yaml \
  --set secrets.mode=reference \
  --set secrets.existingSecretName=my-provider-creds \
  --dependency-update \
  -n confidential-containers-system

cat > providers/gcp-wif-values.yaml <<EOF
# WIF-specific configuration
gcp:
  workloadIdentityFederation:
    enable: true
    serviceAccount: "${GSA_EMAIL}"
    workloadPool: "${GKE_WORKLOAD_POOL}"
    cluster: "${PROVIDER_NAME}"
EOF

helm install peerpods . \
  -f providers/gcp.yaml \
  -f providers/gcp-wif-values.yaml \
  --dependency-update \
  -n confidential-containers-system

kubectl get secret peer-pods-secret \
  -n confidential-containers-system \
  -o jsonpath='{.data.GCP_WIF_CREDENTIALS}' | base64 -d | jq

CAA_POD=$(kubectl get pods -n confidential-containers-system \
  -l app=cloud-api-adaptor \
  -o jsonpath='{.items[0].metadata.name}')

kubectl logs -n confidential-containers-system ${CAA_POD} | grep "GCP_WIF_CREDENTIALS"

Generic Peer pods Helm charts deployment instructions are also described here.

Verify deployment

Follow the shared installation verification. You can then use either sample below.

Run sample application

• CoCo Secret Retrieval
• Basic nginx

# initdata.toml
algorithm = "sha384"
version = "0.1.0"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]
url = 'http://127.0.0.1:8080'

[token_configs.kbs]
url = 'http://127.0.0.1:8080'
cert = """
-----BEGIN CERTIFICATE-----
MIIDljCCAn6gAwIBAgIUR/UNh13GFam4emgludtype/S9BIwDQYJKoZIhvcNAQEL
BQAwdTELMAkGA1UEBhMCQ04xETAPBgNVBAgMCFpoZWppYW5nMREwDwYDVQQHDAhI
YW5nemhvdTERMA8GA1UECgwIQUFTLVRFU1QxFDASBgNVBAsMC0RldmVsb3BtZW50
MRcwFQYDVQQDDA5BQVMtVEVTVC1IVFRQUzAeFw0yNDAzMTgwNzAzNTNaFw0yNTAz
MTgwNzAzNTNaMHUxCzAJBgNVBAYTAkNOMREwDwYDVQQIDAhaaGVqaWFuZzERMA8G
A1UEBwwISGFuZ3pob3UxETAPBgNVBAoMCEFBUy1URVNUMRQwEgYDVQQLDAtEZXZl
bG9wbWVudDEXMBUGA1UEAwwOQUFTLVRFU1QtSFRUUFMwggEiMA0GCSqGSIb3DQEB
AQUAA4IBDwAwggEKAoIBAQDfp1aBr6LiNRBlJUcDGcAbcUCPG6UzywtVIc8+comS
ay//gwz2AkDmFVvqwI4bdp/NUCwSC6ShHzxsrCEiagRKtA3af/ckM7hOkb4S6u/5
ewHHFcL6YOUp+NOH5/dSLrFHLjet0dt4LkyNBPe7mKAyCJXfiX3wb25wIBB0Tfa0
p5VoKzwWeDQBx7aX8TKbG6/FZIiOXGZdl24DGARiqE3XifX7DH9iVZ2V2RL9+3WY
05GETNFPKtcrNwTy8St8/HsWVxjAzGFzf75Lbys9Ff3JMDsg9zQzgcJJzYWisxlY
g3CmnbENP0eoHS4WjQlTUyY0mtnOwodo4Vdf8ZOkU4wJAgMBAAGjHjAcMBoGA1Ud
EQQTMBGCCWxvY2FsaG9zdIcEfwAAATANBgkqhkiG9w0BAQsFAAOCAQEAKW32spii
t2JB7C1IvYpJw5mQ5bhIlldE0iB5rwWvNbuDgPrgfTI4xiX5sumdHw+P2+GU9KXF
nWkFRZ9W/26xFrVgGIS/a07aI7xrlp0Oj+1uO91UhCL3HhME/0tPC6z1iaFeZp8Y
T1tLnafqiGiThFUgvg6PKt86enX60vGaTY7sslRlgbDr9sAi/NDSS7U1PviuC6yo
yJi7BDiRSx7KrMGLscQ+AKKo2RF1MLzlJMa1kIZfvKDBXFzRd61K5IjDRQ4HQhwX
DYEbQvoZIkUTc1gBUWDcAUS5ztbJg9LCb9WVtvUTqTP2lGuNymOvdsuXq+sAZh9b
M9QaC1mzQ/OStg==
-----END CERTIFICATE-----
"""
'''

"cdh.toml"  = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = 'http://1.2.3.4:8080'
kbs_cert = """
-----BEGIN CERTIFICATE-----
MIIFTDCCAvugAwIBAgIBADBGBgkqhkiG9w0BAQowOaAPMA0GCWCGSAFlAwQCAgUA
oRwwGgYJKoZIhvcNAQEIMA0GCWCGSAFlAwQCAgUAogMCATCjAwIBATB7MRQwEgYD
VQQLDAtFbmdpbmVlcmluZzELMAkGA1UEBhMCVVMxFDASBgNVBAcMC1NhbnRhIENs
YXJhMQswCQYDVQQIDAJDQTEfMB0GA1UECgwWQWR2YW5jZWQgTWljcm8gRGV2aWNl
czESMBAGA1UEAwwJU0VWLU1pbGFuMB4XDTIzMDEyNDE3NTgyNloXDTMwMDEyNDE3
NTgyNlowejEUMBIGA1UECwwLRW5naW5lZXJpbmcxCzAJBgNVBAYTAlVTMRQwEgYD
VQQHDAtTYW50YSBDbGFyYTELMAkGA1UECAwCQ0ExHzAdBgNVBAoMFkFkdmFuY2Vk
IE1pY3JvIERldmljZXMxETAPBgNVBAMMCFNFVi1WQ0VLMHYwEAYHKoZIzj0CAQYF
K4EEACIDYgAExmG1ZbuoAQK93USRyZQcsyobfbaAEoKEELf/jK39cOVJt1t4s83W
XM3rqIbS7qHUHQw/FGyOvdaEUs5+wwxpCWfDnmJMAQ+ctgZqgDEKh1NqlOuuKcKq
2YAWE5cTH7sHo4IBFjCCARIwEAYJKwYBBAGceAEBBAMCAQAwFwYJKwYBBAGceAEC
BAoWCE1pbGFuLUIwMBEGCisGAQQBnHgBAwEEAwIBAzARBgorBgEEAZx4AQMCBAMC
AQAwEQYKKwYBBAGceAEDBAQDAgEAMBEGCisGAQQBnHgBAwUEAwIBADARBgorBgEE
AZx4AQMGBAMCAQAwEQYKKwYBBAGceAEDBwQDAgEAMBEGCisGAQQBnHgBAwMEAwIB
CDARBgorBgEEAZx4AQMIBAMCAXMwTQYJKwYBBAGceAEEBEDDhCejDUx6+dlvehW5
cmmCWmTLdqI1L/1dGBFdia1HP46MC82aXZKGYSutSq37RCYgWjueT+qCMBE1oXDk
d1JOMEYGCSqGSIb3DQEBCjA5oA8wDQYJYIZIAWUDBAICBQChHDAaBgkqhkiG9w0B
AQgwDQYJYIZIAWUDBAICBQCiAwIBMKMDAgEBA4ICAQACgCai9x8DAWzX/2IelNWm
ituEBSiq9C9eDnBEckQYikAhPasfagnoWFAtKu/ZWTKHi+BMbhKwswBS8W0G1ywi
cUWGlzigI4tdxxf1YBJyCoTSNssSbKmIh5jemBfrvIBo1yEd+e56ZJMdhN8e+xWU
bvovUC2/7Dl76fzAaACLSorZUv5XPJwKXwEOHo7FIcREjoZn+fKjJTnmdXce0LD6
9RHr+r+ceyE79gmK31bI9DYiJoL4LeGdXZ3gMOVDR1OnDos5lOBcV+quJ6JujpgH
d9g3Sa7Du7pusD9Fdap98ocZslRfFjFi//2YdVM4MKbq6IwpYNB+2PCEKNC7SfbO
NgZYJuPZnM/wViES/cP7MZNJ1KUKBI9yh6TmlSsZZOclGJvrOsBZimTXpATjdNMt
cluKwqAUUzYQmU7bf2TMdOXyA9iH5wIpj1kWGE1VuFADTKILkTc6LzLzOWCofLxf
onhTtSDtzIv/uel547GZqq+rVRvmIieEuEvDETwuookfV6qu3D/9KuSr9xiznmEg
xynud/f525jppJMcD/ofbQxUZuGKvb3f3zy+aLxqidoX7gca2Xd9jyUy5Y/83+ZN
bz4PZx81UJzXVI9ABEh8/xilATh1ZxOePTBJjN7lgr0lXtKYjV/43yyxgUYrXNZS
oLSG2dLCK9mjjraPjau34Q==
-----END CERTIFICATE-----
"""
'''

"policy.rego" = '''
package agent_policy

import future.keywords.in
import future.keywords.every

import input

# Default values, returned by OPA when rules cannot be evaluated to true.
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := false
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default OnlineCPUMemRequest := true
default PullImageRequest := true
default ReadStreamRequest := false
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StatsContainerRequest := true
default TtyWinResizeRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := false
'''

INITDATA=$(cat initdata.toml | gzip | base64 -w0)

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: example-pod
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: "$INITDATA"
spec:
  runtimeClassName: kata-remote
  containers:
    - name: example-container
      image: alpine:latest
      command:
        - sleep
        - "3600"
      securityContext:
        privileged: false
        seccompProfile:
          type: RuntimeDefault
EOF

kubectl exec -it example-pod -- curl http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1

Ensure that the pod is up and running:

kubectl get pods -n default

You can verify that the PodVM was created by running the following command:

gcloud compute instances list

Here you should see the VM associated with the pod used by the example above.

Uninstall

First remove all kata-remote workloads, then verify that all peer-pod VMs are deleted:

Use the following command to list all the peer pod VMs (VMs having prefix podvm) and status.

gcloud compute instances list \
  --filter="name~'podvm.*'" \
  --format="table(name,zone,status)"

After the PodVMs are gone, uninstall CAA.

Remove the GCP service account and its project-level IAM bindings:

gcloud projects remove-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/compute.instanceAdmin.v1"

gcloud projects remove-iam-policy-binding ${GCP_PROJECT_ID} \
  --member="serviceAccount:${GSA_EMAIL}" \
  --role="roles/iam.serviceAccountUser"

gcloud iam service-accounts delete ${GSA_EMAIL} \
  --project=${GCP_PROJECT_ID}

Workload Identity Federation only: Delete the OIDC provider and workload identity pool created during setup:

gcloud iam workload-identity-pools providers delete ${OIDC_PROVIDER_NAME} \
  --location=global \
  --workload-identity-pool=${WI_POOL_NAME} \
  --project=${GCP_PROJECT_ID}

gcloud iam workload-identity-pools delete ${WI_POOL_NAME} \
  --location=global \
  --project=${GCP_PROJECT_ID}

Note: Deleting a pool or provider is a soft delete; GCP retains the resource for 30 days before permanently removing it. During this window the name cannot be reused unless you undelete it.

Delete the GKE cluster and confirm deletion when prompted:

gcloud container clusters delete "${CLUSTER_NAME}" \
  --zone "${GCP_REGION}-a"

Debug SSH connection

Note: SSH connection is available only for debug image, which is built with enabled SSH server and added public key to authorized_keys file. If you want to have SSH access to the image, make sure to build debug image and upload it to Google using above instructions.

After creating debug image with enabled SSH, deploy CoCo with sample pod and use root account to access it:

Note: Remember to add your public key to Google using CLI

1. Export environment variable GCP_PODVM_IP using below code:

   GCP_LATEST_PODVM=$(gcloud compute instances list \
                               --project="${GCP_PROJECT_ID}" \
                               --filter="name ~ ^podvm-" \
                               --sort-by=~creationTimestamp \
                               --limit=1 \
                               --format="value(name)")
   
   export GCP_PODVM_IP=$(gcloud compute instances describe "${GCP_LATEST_PODVM}" \
                           --project="${GCP_PROJECT_ID}" \
                           --zone="$(gcloud compute instances list \
                                       --project="${GCP_PROJECT_ID}" \
                                       --filter="name=${GCP_LATEST_PODVM}" \
                                       --format="value(zone)")" \
                           --format="value(networkInterfaces[0].accessConfigs[0].natIP)")
   

2. Connect to debug image using SSH:

   ssh -i ./gcp_ssh_debug root@"$GCP_PODVM_IP"
   

Troubleshooting

Note: If your case is not covered below, see the troubleshooting guide.

VM Doesn’t Start

Starting with GKE version 1.27, GCP configures containerd with the discard_unpacked_layers=true flag to optimize disk usage by removing compressed image layers after they are unpacked. However, this can cause issues with PeerPods, as the workload may fail to locate required layers. To avoid this, disable the discard_unpacked_layers setting in the containerd configuration.

Most of the time you will see a generic message such as the following:

Error: failed to create containerd container: error unpacking image: failed to extract layer sha256:<SHA>: failed to get reader from content store: content digest sha256:<SHA>: not found

To disable the discard_unpacked_layers setting in the containerd configuration on Google Kubernetes Engine (GKE) version 1.27 or later, follow these steps:

1. SSH to worker node Google console

2. Run command which will change the discard_unpacked_layers property to false in the containerd configuration file:

   sudo sed -i 's/discard_unpacked_layers = true/discard_unpacked_layers = false/' /etc/containerd/config.toml
   

3. Verify changed property:

   sudo cat /etc/containerd/config.toml | grep discard_unpacked_layers
   

4. Restart containerd using below command:

   sudo systemctl restart containerd