This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Features

Primitives provided by Confidential Containers

1: Get Attestation
2: Get Secret Resources
3: Signed Images
4: Encrypted Images
5: Authenticated Registries
6: Sealed Secrets
7: Init-Data
8: Image Pull Proxy
9: Local Registries
10: Protected Storage

10.1: Confidential EmptyDir

11: Runtime Attestation

In addition to running pods inside of enclaves, Confidential Containers provides several other features that can be used to protect workloads and data. Securing complex workloads often requires using some of these features.

Most features depend on and require attestation, which is described in the next section.

1 - Get Attestation

Workloads that request attestation evidence

Warning

This feature is disabled by default due to some subtle security considerations which are described below.

Workloads can directly request attestation evidence. A workload could use this evidence to carry out its own attestation protocol.

Enabling

To enable this feature, set the following parameter in the guest kernel command line.

agent.guest_components_rest_api=all

As usual, command line configurations can be added with annotations.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        io.katacontainers.config.hypervisor.kernel_params: "agent.guest_components_rest_api=all"
    spec:
      runtimeClassName: # (...)
      containers:
      - name: nginx
        # (...)

Attestation

Once enabled, an attestation can be retrieved via the REST API.

curl http://127.0.0.1:8006/aa/evidence?runtime_data=xxxx

The API expects runtime data to be provided. The runtime data will be included in the report. Typically this is used to bind a nonce or the hash of a key to the evidence.

Security

Enabling this feature can make your workload vulnerable to so-called “evidence factory” attacks. By default, different CoCo workloads can have the same TCB, even if they use different container images. This is a design feature, but it means that when the attestation report API is enabled, two workloads could produce interchangeable attestation reports, even if they are operated by different parties.

To make sure someone else cannot generate evidence that could be used with your attestation protocol, configure your guest to differentiate its TCB. For example, use the init-data (which is measured) to specify the public key of the KBS. Init-data is currently only supported with peer pods. More information about it will be added soon.

2 - Get Secret Resources

Workloads that request resources from Trustee

Note

Requesting secret resources depends on attestation. Configure attestation before requesting resources

While sealed secrets can be used to store encrypted secret resources as Kubernetes Secrets and have them transparently decrypted using keys from Trustee, a workload can also explicitly request resources via a REST API that is automatically exposed to the pod network namespace.

For example, you can run this command from your container.

curl http://127.0.0.1:8006/cdh/resource/default/key/1

In this example Trustee will fulfill the request for kbs:///default/key/1 assuming that the resource has been provisioned.

3 - Signed Images

Procedures to generate and deploy signed OCI images with CoCo

Overview

Encrypted images provide confidentiality, but they do not provide authenticity or integrity. Image signatures provide this additional property, preventing certain types of image tampering, for example.

In this brief guide, we show two tools that can be used to sign container images: cosign and skopeo. The skopeo tool can be used to create both cosign signatures or “simple signatures” (which leverage gpg keys). For our purposes, our skopeo examples will use the simple signing approach.

In any case, the general approach is to

Create keys for signing,
Sign a newly tagged image, and
Update the KBS with the public signature key and a security policy.

Creating an Image

Creating a Key Pair

Create a key pair using one of two approaches: cosign or simple signing with gpg.

To generate a public/private key pair with cosign, set COSIGN_PASSWORD and run generate-key-pair:

COSIGN_PASSWORD=just1testing2password3 cosign generate-key-pair

This will create the private and public keys: cosign.key and cosign.pub.

skopeo depends on gpg for key management. To generate a key pair with gpg using the default options, run:

gpg --full-generate-key

You will be prompted for several values. For testing, you can use:

GitHub Runner
git@runner.com
just1testing2password3

Then export the key material. The --export-secret-key option is sufficient to export both the private and public keys. For example:

gpg --export-secret-key F63DB2A1AB7C7F195F698C9ED9582CADF7FBCC5D > github-runner.keys

You can later import the keys in a CI system by using --batch to avoid interactive prompts:

gpg --batch --import ./github-runner.keys

When automating CI or test workflows, you can place the key password in a plain-text file when that is acceptable for your environment:

echo just1testing2password3 > git-runner-password.txt

Signing the Image

Sign the image using one of two approaches: cosign or simple signing with skopeo.

In this example, we use a sample minimal Dockerfile to build an image that will be signed.

Create Dockerfile:

cat <<EOF > Dockerfile
FROM nginx:1.27-alpine

EXPOSE 80
EOF

The workflow is to build the image, push it to ghcr, and then sign it.

Make sure you are authenticated to ghcr first, for example with docker login.

Perform the following steps to build, push, and sign the image:

Build the image

COCO_PKG=confidential-containers/test-container
docker build \
  -t ghcr.io/${COCO_PKG}:cosign-sig \
  -f Dockerfile \
  .

Push the image to ghcr
```
docker push ghcr.io/${COCO_PKG}:cosign-sig
```
After pushing the image, note the image digest shown in the output. You will use it in the signing command. For example:
```
cosign-sig: digest: sha256:<IMAGE_DIGEST> size: 1989
```
Check your cosign version:
```
cosign version
```
Use the signing command that matches your installed version.

cosign `v3.0.x`

cosign sign --new-bundle-format=false \
  --use-signing-config=false \
  --key ./cosign.key \
  ghcr.io/${COCO_PKG}@sha256:<IMAGE_DIGEST>

cosign `>= v2.2.0` and `< v3.0`

cosign sign --key ./cosign.key ghcr.io/${COCO_PKG}@sha256:<IMAGE_DIGEST>

Note

cosign versions v2.0.x and v2.1.x use a different private key format from >= v2.2.0.

Ensure that you have a gpg key owned by the user signing the image. See the previous subsection for instructions on generating and importing gpg keys.

The following example signs a local image named confidential-containers/test-container. It uses the unsigned tag and, as part of the signing flow, creates a new simple-signed tag. In this example, the resulting image is pushed to ghcr, which requires docker login first:

COCO_PKG=confidential-containers/test-container
skopeo \
  copy \
  --debug \
  --insecure-policy \
  --sign-by git@runner.com \
  --sign-passphrase-file ./git-runner-password.txt \
  docker-daemon:ghcr.io/${COCO_PKG}:unsigned \
  docker://ghcr.io/${COCO_PKG}:simple-signed

Running an Image

Running a workload with a signed image is very similar to running a workload with an unsigned image. The main difference is that, for a signed image, you must provide the KBS with the public key and a security policy.

The security policy tells KBS which image is signed, which signature type is used, and where to find the public key that should be used for verification. After that, you can run the workload as usual, for example with kubectl apply.

Setting the Security Policy for Signed Images

Export the KbsConfig custom resource name:

export CR_NAME=$(kubectl get kbsconfig -n trustee-operator-system -o=jsonpath='{.items[0].metadata.name}')

Create a Secret containing the public key:

kubectl create secret generic sig-public-key \
  -n trustee-operator-system \
  --from-file=test=./cosign.pub

Patch the KbsConfig custom resource to add the public key Secret:

kubectl patch KbsConfig -n trustee-operator-system $CR_NAME \
  --type=json \
  -p='[{"op":"add", "path":"/spec/kbsSecretResources/-", "value":"sig-public-key"}]'

Run the following command to add the public key to KBS storage:

./kbs-client --url <SCHEME>://<HOST>:<PORT> config \
  --auth-private-key private.key \
  set-resource \
  --resource-file cosign.pub \
  --path default/sig-public-key/test

Create an image pull validation policy file. For example, create security-policy.json with the following contents:

{
  "default": [
    {
      "type": "reject"
    }
  ],
  "transports": {
    "<transport>": {
      "<registry>/<image>": [
        {
          "type": "sigstoreSigned",
          "keyPath": "kbs:///default/<type>/<tag>"
        }
      ]
    }
  }
}

By default, the policy rejects all images and all signatures. The transports section specifies which images the policy explicitly approves and verifies through their signatures.

Replace placeholders in the policy file with the appropriate values:

<transport> - Specify the image repository for transport, for example, docker. More information can be found in containers-transports 5.
<registry>/<image> - Specify the container registry and image, for example, ghcr.io/confidential-containers/test-container.
<type>/<tag> - Specify the type and tag of the container image signature verification secret that you created, for example, sig-public-key/test.

Finally, register the image pull validation policy file in KBS storage:

Export the KbsConfig custom resource name:

export CR_NAME=$(kubectl get kbsconfig -n trustee-operator-system -o=jsonpath='{.items[0].metadata.name}')

Create a Secret containing the security policy:

kubectl create secret generic security-policy \
  -n trustee-operator-system \
  --from-file=test=./security-policy.json

Patch the KbsConfig custom resource to add the security policy Secret:

kubectl patch KbsConfig -n trustee-operator-system $CR_NAME \
  --type=json \
  -p='[{"op":"add", "path":"/spec/kbsSecretResources/-", "value":"security-policy"}]'

Run the following command to add the security policy to KBS storage:

./kbs-client --url <SCHEME>://<HOST>:<PORT> config \
  --auth-private-key private.key \
  set-resource \
  --resource-file ./security-policy.json \
  --path default/security-policy/test

Enable Signature Verification

To enforce signature verification for a Pod, you must set the appropriate kernel parameters or init data in the pod annotation.

Set the following kernel parameters in the io.katacontainers.config.hypervisor.kernel_params pod annotation:

agent.image_policy_file=kbs:///default/<SECRET_POLICY_NAME>/<KEY> agent.enable_signature_verification=true agent.aa_kbc_params=cc_kbc::<SCHEME>://<HOST>:<PORT>

agent.image_policy_file points to the security policy file registered in KBS storage. In this example, SECRET_POLICY_NAME = security-policy and KEY = test.
agent.enable_signature_verification enables signature verification.
agent.aa_kbc_params points to the KBS service. Replace <SCHEME>, <HOST>, and <PORT> with the values for your environment.

Run the following commands to prepare the init data file with the appropriate KBS configuration and security policy:

Export environment variables:

export KBS_ADDRESS=scheme://host:port
export SECRET_POLICY_NAME=security-policy
export SECRET_POLICY_KEY=test

Create file $HOME/initdata.toml
```
cat <<EOF> initdata.toml
algorithm = "sha256"
version = "0.1.0"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]
url = '${KBS_ADDRESS}'

[token_configs.kbs]
url = '${KBS_ADDRESS}'
'''

"cdh.toml" = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '${KBS_ADDRESS}'

[image]
image_security_policy_uri = 'kbs:///default/${SECRET_POLICY_NAME}/${SECRET_POLICY_KEY}'
'''
EOF
```
The most important fields in the [image] section are:
- image_security_policy_uri - Points to the image security policy that image-rs uses when pulling images. Commonly a KBS URI such as kbs:///default/..., but it can also point to a local file, for example file:///etc/image-policy.json. If this field is not set, no image security policy is applied and pulled images are not validated.
- image_security_policy - Inline alternative that lets you provide the policy content directly as a string instead of referencing it through a URI. If both image_security_policy_uri and image_security_policy are set, image_security_policy_uri takes precedence.

Encode the init data file in base64:

export INIT_DATA=$(cat $HOME/initdata.toml | gzip | base64 -w 0)

Set the output from above command in the io.katacontainers.config.hypervisor.cc_init_data pod annotation:

io.katacontainers.config.hypervisor.cc_init_data = ${INIT_DATA}

Run a Signed Workload

Create a Pod that uses the signed image.

Replace <SCHEME>, <KBS_HOST>, <KBS_PORT>, and runtimeClassName with the values appropriate for your environment:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: test-container
  name: test-container
  annotations:
    io.katacontainers.config.hypervisor.kernel_params: agent.aa_kbc_params=cc_kbc::<SCHEME>://<KBS_HOST>:<KBS_PORT> agent.image_policy_file=kbs:///default/security-policy/test agent.enable_signature_verification=true
spec:
  containers:
    - name: test-container
      image: ghcr.io/confidential-containers/test-container:cosign-sig
  dnsPolicy: ClusterFirst
  runtimeClassName: <RUNTIME_CLASS>
EOF

Verify that the Pod is running:

$ kubectl get pod test-container -o jsonpath='{.status.phase}{"\n"}'
Running

Troubleshooting

If image signature verification fails, you may see an error similar to the following in the Pod log:

Image Pull error: Failed to pull image [IMAGE] from all mirror/mapping locations or original location: \
image: [IMAGE], error: Image policy rejected: Denied by policy: rejected by `sigstoreSigned` rule

In that case, check the following:

Ensure that the public key is correctly registered in KBS storage and that the key path in the security policy file is correct.
Ensure that the security policy file is correctly registered in KBS storage and that the path in the pod annotation is correct.
Ensure that the image is correctly signed and that the signature is valid.

4 - Encrypted Images

Procedures to encrypt and consume OCI images in a TEE

Context

A user might want to bundle sensitive data on an OCI (Docker) image. The image layers should only be accessible within a Trusted Execution Environment (TEE).

The project provides the means to encrypt an image with a symmetric key that is released to the TEE only after successful verification and appraisal in a Remote Attestation process. CoCo infrastructure components within the TEE will transparently decrypt the image layers as they are pulled from a registry without exposing the decrypted data outside the boundaries of the TEE.

Instructions

The following steps require a functional CoCo installation on a Kubernetes cluster. A Key Broker Client (KBC) has to be configured for TEEs to be able to retrieve confidential secrets. We assume cc_kbc as a KBC for the CoCo project’s Key Broker Service (KBS) in the following instructions, but image encryption should work with other Key Broker implementations in a similar fashion.

Note

Please ensure you have a recent version of Skopeo (v1.13.3+) installed locally.

Encrypt an image

We extend public image with secret data.

docker build -t unencrypted - <<EOF
FROM nginx:stable
RUN echo "something confidential" > /secret
EOF

The encryption key needs to be a 32 byte sequence and provided to the encryption step as base64-encoded string.

KEY_FILE="image_key"
head -c 32 /dev/urandom | openssl enc > "$KEY_FILE"
KEY_B64="$(base64 < $KEY_FILE)"

The key id is a generic resource descriptor used by the key broker to look up secrets in its storage. For KBS this is composed of three segments: $repository_name/$resource_type/$resource_tag

KEY_PATH="/default/image_key/nginx"
KEY_ID="kbs://${KEY_PATH}"

The image encryption logic is bundled and invoked in a container:

git clone https://github.com/confidential-containers/guest-components.git
cd guest-components
docker build -t coco-keyprovider -f ./attestation-agent/docker/Dockerfile.keyprovider .

To access the image from within the container, Skopeo can be used to buffer the image in a directory, which is then made available to the container. Similarly, the resulting encrypted image will be put into an output directory.

mkdir -p oci/{input,output}
skopeo copy docker-daemon:unencrypted:latest dir:./oci/input
docker run -v "${PWD}/oci:/oci" coco-keyprovider /encrypt.sh -k "$KEY_B64" -i "$KEY_ID" -s dir:/oci/input -d dir:/oci/output

We can inspect layer annotations to confirm the expected encryption was applied:

skopeo inspect dir:./oci/output | jq '.LayersData[0].Annotations["org.opencontainers.image.enc.keys.provider.attestation-agent"] | @base64d | fromjson'

Sample output:

{
  "kid": "kbs:///default/image_key/nginx",
  "wrapped_data": "lGaLf2Ge5bwYXHO2g2riJRXyr5a2zrhiXLQnOzZ1LKEQ4ePyE8bWi1GswfBNFkZdd2Abvbvn17XzpOoQETmYPqde0oaYAqVTMcnzTlgdYYzpWZcb3X0ymf9bS0gmMkqO3dPH+Jf4axXuic+ITOKy7MfSVGTLzay6jH/PnSc5TJ2WuUJY2rRtNaTY65kKF2K9YP6mtYBqcHqvPDlFiVNNeTAGv2w1zwaMlgZaSHV+Z1y+xxbOV5e98bxuo6861rMchjCiE7FY37PHD3a5ISogq90=",
  "iv": "Z8bGQL7r6qxSpd4L",
  "wrap_type": "A256GCM"
}

Finally, the resulting encrypted image can be provisioned to an image registry.

ENCRYPTED_IMAGE=some-private.registry.io/coco/nginx:encrypted
skopeo copy dir:./oci/output "docker://${ENCRYPTED_IMAGE}"

Provision image key

Prior to launching a Pod the image key needs to be provisioned to the Key Broker’s repository. For a KBS deployment on Kubernetes using the local filesystem as repository storage it would work like this:

kubectl exec deploy/kbs -- mkdir -p "/opt/confidential-containers/kbs/repository/$(dirname "$KEY_PATH")"
cat "$KEY_FILE" | kubectl exec -i deploy/kbs -- tee "/opt/confidential-containers/kbs/repository/${KEY_PATH}" > /dev/null

Note: If you’re not using KBS deployment using trustee operator additional namespace may be needed -n coco-tenant.

Launch a Pod

We create a simple deployment using our encrypted image. As the image is being pulled and the CoCo components in the TEE encounter the layer annotations that we saw above, the image key will be retrieved from the Key Broker using the annotated Key ID and the layers will be decrypted transparently and the container should come up.

In this example we default to the Cloud API Adaptor runtime, adjust this depending on the CoCo installation.

kubectl get runtimeclass -o jsonpath='{.items[].handler}'

Sample output:

kata-remote

Export variable:

CC_RUNTIMECLASS=kata-remote

Export KBS address:

KBS_ADDRESS=scheme://host:port

Deploy sample pod:

cat <<EOF> nginx-encrypted.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-encrypted
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        io.katacontainers.config.hypervisor.kernel_params: "agent.aa_kbc_params=cc_kbc::${KBS_ADDRESS}"
        io.containerd.cri.runtime-handler: ${CC_RUNTIMECLASS}
    spec:
      runtimeClassName: ${CC_RUNTIMECLASS}
      containers:
      - image: ${ENCRYPTED_IMAGE}
        name: nginx
        imagePullPolicy: Always
EOF
kubectl apply -f nginx-encrypted.yaml

Create file $HOME/initdata.toml

cat <<EOF> initdata.toml
algorithm = "sha256"
version = "0.1.1"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]
url = '${KBS_ADDRESS}'

[token_configs.kbs]
url = '${KBS_ADDRESS}'
'''

"cdh.toml"  = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '${KBS_ADDRESS}'
'''
EOF

Export variable:

INIT_DATA_B64=$(cat $HOME/initdata.toml | gzip | base64 -w0)

Deploy:

cat <<EOF> nginx-encrypted.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-encrypted
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        io.katacontainers.config.hypervisor.cc_init_data: "${INIT_DATA_B64}"
        io.containerd.cri.runtime-handler: ${CC_RUNTIMECLASS}
    spec:
      runtimeClassName: ${CC_RUNTIMECLASS}
      containers:
      - image: ${ENCRYPTED_IMAGE}
        name: nginx
        imagePullPolicy: Always
EOF
kubectl apply -f nginx-encrypted.yaml

We can confirm that the image key has been retrieved from KBS.

$ kubectl logs -f deploy/kbs | grep "$KEY_PATH"
[2024-01-23T10:24:52Z INFO  actix_web::middleware::logger] 10.244.0.1 "GET /kbs/v0/resource/default/image_key/nginx HTTP/1.1" 200 530 "-" "attestation-agent-kbs-client/0.1.0" 0.000670

Note: If you’re not using KBS deployment using trustee operator additional namespace may be needed -n coco-tenant.

Debugging

The encrypted image feature relies on a cascade of other features as building blocks:

Image decryption is built on secret retrieval from KBS
Secret retrieval is built on remote attestation
Remote attestation requires retrieval of hardware evidence from the TEE

All of the above have to work in order to decrypt an encrypted image.

0. Launching with an unencrypted image

Launch the same image with unencrypted layers from the same registry to verify that the image itself is not an issue.

1. Retrieve hardware evidence from TEE

Launch an unencrypted library/nginx deployment named nginx with a CoCo runtime class. Issue kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/aa/evidence\?runtime_data\=xxxx. This should produce a real hardware evidence rendered as JSON. Something like {"svn":"1","report_data":"eHh4eA=="} is not real hardware evidence, it’s a dummy response from the sample attester. Real TEE evidence should be more verbose and contain certificates and measurements.

The reason for not producing real evidence could be a wrong build of Attestation Agent for the TEE that you are attempting to use, or the Confidential VM not exposing the expected interfaces.

Note: In some configurations of the CoCo image the facility to retrieve evidence is disabled by default. For bare-metal CoCo images you can enable it by setting agent.guest_components_rest_api=all on the kernel cmdline (see here).

2. Perform remote attestation

Run kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/aa/token\?token_type\=kbs. This should produce an attestation token. If you don’t receive a token but an error you should inspect your Trustee KBS/AS logs and see whether there was a connection attempt for the CVM and potentially a reason why remote attestation failed.

You can set RUST_LOG=debug as environment variable the Trustee deployment to receive more verbose logs. If the evidence is being sent to KBS, the issue is most likely resolvable on the KBS side and possibly related to the evidence not meeting the expectations for a key release in KBS/AS policies or the requested resource not being available in the KBS.

If you don’t see an attestation attempt in the log of KBS there might be problems with network connectivity between the Confidential VM and the KBS. Note that the Pod and the Guest Components on the CVM might not share the same network namespace. A Pod might be able to reach KBS, but the Attestation Agent on the Confidential VM might not. If you are using HTTPS to reach the KBS, there might be a problem with the certificate provided to the Confidential VM (e.g via Initdata).

3. Retrieve decryption key from KBS

If you have successfully retrieved a token attempt to fetch the symmetric key for the encrypted image, manually and using the image’s key id (kid): kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/cdh/resource/default/images/my-key.

You can find out the kid for a given encrypted image in the a query like this:

$ ANNOTATION="org.opencontainers.image.enc.keys.provider.attestation-agent"
$ skopeo inspect docker://ghcr.io/mkulke/nginx-encrypted@sha256:5a81641ff9363a63c3f0a1417d29b527ff6e155206a720239360cc6c0722696e \
    | jq --arg ann "$ANNOTATION" -r '.LayersData[0].Annotations[$ann] | @base64d | fromjson | .kid'
kbs:///default/image_key/nginx

If the key can be retrieved successfully, verify that the size is exactly 32 bytes and matches the key you used for encrypting the image.

4. Other (desperate) measures

If pulling an encrypted image still doesn’t work after successful retrieval of its encryption key, you might want to purge the node(s). There are bugs in the containerd remote snapshotter implementation that might taint the node with manifests and layers that will interfere with image pulling. The node should be discarded and reinstalled to rule that out.

It might also help to set annotations.io.containerd.cri.runtime-handler: $your-runtime-class in the pod spec, in addition to the runtimeClassName field and ImagePullPolicy: Always to ensure that the image is always pulled from the registry.

5 - Authenticated Registries

Use private OCI registries

Context

A user might want to use container images from private OCI registries, hence requiring authentication. The project provides the means to pull protected images from authenticated registry.

Important: ideally the authentication credentials should only be accessible from within a Trusted Execution Environment, however, due some limitations on the architecture of components used by CoCo, the credentials need to be exposed to the host, thus registry authentication is not currently a confidential feature. The community has worked to remediate that limitation and, in meanwhile, we recommend the use of encrypted images as a mitigation.

Instructions

The following steps require a functional CoCo installation on a Kubernetes cluster. A Key Broker Client (KBC) has to be configured for TEEs to be able to retrieve confidential secrets. We assume cc_kbc as a KBC for the CoCo project’s Key Broker Service (KBS) in the following instructions, but authenticated registries should work with other Key Broker implementations in a similar fashion.

Create registry authentication file

The registry authentication file should have the containers-auth.json format, with exception of credential helpers (credHelpers) that aren’t supported. Also it’s not supported glob URLs nor prefix-matched paths as in Kubernetes interpretation of config.json.

Create the registry authentication file (e.g containers-auth.json) like this:

export AUTHENTICATED_IMAGE="my-registry.local/repository/image:latest"
export AUTHENTICATED_IMAGE_NAMESPACE="$(echo "$AUTHENTICATED_IMAGE" | cut -d':' -f1)"
export AUTHENTICATED_IMAGE_USER="MyRegistryUser"
export AUTHENTICATED_IMAGE_PASSWORD="MyRegistryPassword"
cat <<EOF>> containers-auth.json
{
	"auths": {
		"${AUTHENTICATED_IMAGE_NAMESPACE}": {
			"auth": "$(echo ${AUTHENTICATED_IMAGE_USER}:${AUTHENTICATED_IMAGE_PASSWORD} | base64 -w 0)"
		}
	}
}
EOF

Where:

AUTHENTICATED_IMAGE is the full-qualified image name
AUTHENTICATED_IMAGE_NAMESPACE is the image name without the tag
AUTHENTICATED_IMAGE_USER and AUTHENTICATED_IMAGE_PASSWORD are the registry credentials user and password, respectively
auth’s value is the colon-separated user and password (user:password) credentials string encoded in base64

Provision the registry authentication file

Prior to launching a Pod the registry authentication file needs to be provisioned to the Key Broker’s repository. For a KBS deployment on Kubernetes using the local filesystem as repository storage it would work like this:

export KEY_PATH="default/containers/auth"
kubectl exec deploy/kbs -c kbs -n coco-tenant -- mkdir -p "/opt/confidential-containers/kbs/repository/$(dirname "$KEY_PATH")"
cat containers-auth.json | kubectl exec -i deploy/kbs -c kbs -n coco-tenant -- tee "/opt/confidential-containers/kbs/repository/${KEY_PATH}" > /dev/null

The CoCo infrastructure components need to cooperate with containerd and nydus-snapshotter to pull the container image from TEE. Currently the nydus-snapshotter needs to fetch the image’s metadata from registry, then authentication credentials are read from a Kubernetes secret of docker-registry type. So it should be created a secret like this:

export SECRET_NAME="cococred"
kubectl create secret docker-registry "${SECRET_NAME}" --docker-server="https://${AUTHENTICATED_IMAGE_NAMESPACE}" \
    --docker-username="${AUTHENTICATED_IMAGE_USER}" --docker-password="${AUTHENTICATED_IMAGE_PASSWORD}"

Where:

SECRET_NAME is any secret name

Launch a Pod

Create the pod yaml (e.g. pod-image-auth.yaml) like below and apply it:

export KBS_ADDRESS="172.18.0.3:31731"
export RUNTIMECLASS="kata-qemu-coco-dev"
cat <<EOF>> pod-image-auth.yaml
apiVersion: v1
kind: Pod
metadata:
  name: image-auth-feat
  annotations:
    io.containerd.cri.runtime-handler: ${RUNTIMECLASS}
    io.katacontainers.config.hypervisor.kernel_params: ' agent.image_registry_auth=kbs:///${KEY_PATH} agent.guest_components_rest_api=resource agent.aa_kbc_params=cc_kbc::http://${KBS_ADDRESS}'
spec:
  runtimeClassName: ${RUNTIMECLASS}
  containers:
    - name: test-container
      image: ${AUTHENTICATED_IMAGE}
      imagePullPolicy: Always
      command:
        - sleep
        - infinity
  imagePullSecrets:
    - name: ${SECRET_NAME}
EOF

Where:

KBS_ADDRESS is the host:port address of KBS
RUNTIMECLASS is any of available CoCo runtimeclasses (e.g. kata-qemu-tdx, kata-qemu-snp). For this example, kata-qemu-coco-dev allows to create CoCo pod on systems without confidential hardware. It should be replaced with a class matching the TEE in use.

What distinguish the pod specification for authenticated registry from a regular CoCo pod is:

the agent.image_registry_auth property in io.katacontainers.config.hypervisor.kernel_params annotation indicates the location of the registry authentication file as a resource in the KBS
the imagePullSecrets as required by nydus-snapshotter

Check the pod gets Running:

$ kubectl get -f pod-image-auth.yaml
NAME              READY   STATUS    RESTARTS   AGE
image-auth-feat   1/1     Running   0          2m52s

6 - Sealed Secrets

Create, sign, and deploy protected Kubernetes secrets for confidential workloads

Note

Sealed Secrets depend on attestation. Configure attestation before using sealed secrets.

In Confidential Containers, secrets can be protected with sealing. A sealed secret is a way to encapsulate confidential data such that it can be accessed only inside an enclave in conjunction with attestation. At the same time, the untrusted control plane can still store and orchestrate it like a normal Kubernetes secret. Once unsealed inside the guest, the secret can be transparently provisioned to your workload as an environment variable or a volume.

Basic Usage

Here’s how you create a vault secret. There are also envelope secrets, which are described later. Vault secrets are a pointer to resource stored in a KBS, while envelope secrets are wrapped secrets that are unwrapped with a KMS.

Creating a sealed secret

You need a signing key in JWK format with a P256 EC key.
You can generate one with any tool that supports JWK, such as mkjwk.
Show steps to generate JWK using `mkjwk` site

Steps to generate a signing key with mkjwk:
1. Go to mkjwk
2. Choose EC tab
3. Set Curve to P-256
4. Set Key Use to Signature
5. Set Algorithm to ES256
6. Set Key ID (specified) and provide a value
7. Set Show X.509 to No
8. Click Generate button
After you generate the key, you can copy the JWK keypair.

Sample public and private key in JWK format referenced later as private_public_jwk.json:
```
{
    "kty": "EC",
    "d": "TyQ1w0-ZZQUVOa7bkYg_Tlkn18oiPInodQrxQeXUIys",
    "use": "sig",
    "crv": "P-256",
    "kid": "my-kid",
    "x": "FMd3htSXI0dQBo-HfUif2lqShS-47AiQSlJFLnzgXTU",
    "y": "yXuoRnBOuGggB6OogUjDuz3STjy-zS1XREOTF69rEdI",
    "alg": "ES256"
}
```
Sample corresponding public key in JWK format referenced later as public_jwk.json:
```
{
    "kty": "EC",
    "use": "sig",
    "crv": "P-256",
    "kid": "my-kid",
    "x": "FMd3htSXI0dQBo-HfUif2lqShS-47AiQSlJFLnzgXTU",
    "y": "yXuoRnBOuGggB6OogUjDuz3STjy-zS1XREOTF69rEdI",
    "alg": "ES256"
}
```
Use a helper CLI tool for sealed secrets which is available in the Guest Components repository.
```
git clone https://github.com/confidential-containers/guest-components.git
cd guest-components
cargo run -p confidential-data-hub --bin secret --help
```
With the tool you can create a secret.
```
SIGNING_KID_KBS_URI=default/test_signing/jwk_public
SIGNING_JWK_PATH=./path/to/private_public_jwk.json
SIGNING_RESOURCE_URI=default/test_secrets/your_secret

export POINTER_TO_SECRET=$(cargo run -p confidential-data-hub --bin secret seal \
    --signing-kid kbs:///${SIGNING_KID_KBS_URI} --signing-jwk-path ${SIGNING_JWK_PATH} \
    vault \
    --resource-uri kbs:///${SIGNING_RESOURCE_URI} --provider kbs | grep -v "Warning")
echo ${POINTER_TO_SECRET}
```
Key options:
- --signing-kid: the identifier embedded in the JWS header. When this is a Trustee resource URI, the guest can fetch the public key during verification.
- --signing-jwk-path: path to a JWK containing the private signing key.
- vault: selects the vault secret format.
- --resource-uri: provider-specific location of the actual secret.
- --provider: the provider that will resolve the secret, such as kbs.
This command should return a base64 string ( see Integrity protection and signing) which you will use in the next step.

Note

For vault secrets, the secret-cli tool does not upload your resource to the KBS automatically. In addition to generating the secret string, you must also upload the public key (JWK) and resource to your KBS.

Adding a sealed secret to Kubernetes

Create a secret from your secret string using kubectl.

kubectl create secret generic sealed-secret --from-literal=secret=${POINTER_TO_SECRET}

When using --from-literal you provide a mapping of secret keys and values. The secret value should be the string generated in the previous step. The secret key can be whatever you want, but make sure to use the same one in future steps. This is separate from the name of the secret.

Configuring the KBS for your workload

Upload the secret value to the KBS

Create file with your secret value (which will be returned when the sealed secret is unsealed).

cat >> my_secret << 'END'
my-secret-value
END

Upload the secret to the KBS at the resource URI specified in the --resource-uri parameter when you created the sealed secret.

Run the following command to add the secret to KBS storage:

./kbs-client --url <SCHEME>://<HOST>:<PORT> config \
  --auth-private-key private.key \
  set-resource \
  --resource-file ./my_secret \
  --path ${SIGNING_RESOURCE_URI}

Export the KbsConfig custom resource name:

export CR_NAME=$(kubectl get kbsconfig -n trustee-operator-system -o=jsonpath='{.items[0].metadata.name}')

Create a Secret containing your secret value:

kubectl create secret generic test_secrets \
  -n trustee-operator-system \
  --from-file=your_secret=./my_secret

Patch the KbsConfig custom resource to add the public key Secret:

kubectl patch KbsConfig -n trustee-operator-system $CR_NAME \
  --type=json \
  -p='[{"op":"add", "path":"/spec/kbsSecretResources/-", "value":"test_secrets"}]'

Upload the JWK public key to the KBS

This is needed for the guest to verify the signature of the sealed secret.

Create file with your JWK public key.

cat >> public_jwk.json << 'END'
{
    "kty": "EC",
    "use": "sig",
    "crv": "P-256",
    "kid": "my-kid",
    "x": "FMd3htSXI0dQBo-HfUif2lqShS-47AiQSlJFLnzgXTU",
    "y": "yXuoRnBOuGggB6OogUjDuz3STjy-zS1XREOTF69rEdI",
    "alg": "ES256"
}
END

Upload the JWK to the KBS at the resource URI specified in the --signing-kid parameter when you created the sealed secret.

Run the following command to add the public key to KBS storage:

./kbs-client --url <SCHEME>://<HOST>:<PORT> config \
  --auth-private-key kbs.key \
  set-resource \
  --resource-file ./public_jwk.json \
  --path ${SIGNING_KID_KBS_URI}

Export the KbsConfig custom resource name:

export CR_NAME=$(kubectl get kbsconfig -n trustee-operator-system -o=jsonpath='{.items[0].metadata.name}')

Create a Secret containing the JWK public key:

kubectl create secret generic test_signing \
  -n trustee-operator-system \
  --from-file=jwk_public=./public_jwk.json

Patch the KbsConfig custom resource to add the JWK public key secret:

kubectl patch KbsConfig -n trustee-operator-system $CR_NAME \
  --type=json \
  -p='[{"op":"add", "path":"/spec/kbsSecretResources/-", "value":"test_signing"}]'

Deploying a sealed secret to a confidential workload

In order for your workload to use the sealed secret, the CDH inside the guest needs to know how to retrieve the plaintext secret value from the KBS.

Perform below steps to configure the CDH and reference the sealed secret from your workload.

Choose either of the following approaches to provide KBS configuration to the guest:

Set the following kernel parameters in the io.katacontainers.config.hypervisor.kernel_params pod annotation:

agent.aa_kbc_params=cc_kbc::<SCHEME>://<HOST>:<PORT>

- `agent.aa_kbc_params` points to the KBS service. Replace `<SCHEME>`, `<HOST>`, and `<PORT>` with the values for
  your environment.

Run the following commands to prepare the init data file with the appropriate KBS configuration:

- Export environment variables:

  ```bash
  export KBS_ADDRESS=scheme://host:port
  ```

- Create file `$HOME/initdata.toml`
   ```bash
   cat <<EOF> initdata.toml
   algorithm = "sha256"
   version = "0.1.0"
   
   [data]
   "aa.toml" = '''
   [token_configs]
   [token_configs.coco_as]
   url = '${KBS_ADDRESS}'
   
   [token_configs.kbs]
   url = '${KBS_ADDRESS}'
   '''
   
   "cdh.toml" = '''
   socket = 'unix:///run/confidential-containers/cdh.sock'
   credentials = []
   
   [kbc]
   name = 'cc_kbc'
   url = '${KBS_ADDRESS}'
   '''
   EOF
   ```

- Encode the init data file in base64:

  ```bash
  export INIT_DATA=$(cat $HOME/initdata.toml | gzip | base64 -w 0)
  ```

Set the output from above command in the io.katacontainers.config.hypervisor.cc_init_data pod annotation:

io.katacontainers.config.hypervisor.cc_init_data = ${INIT_DATA}

Reference the sealed secret from your workload. You can reference the secret as either an environment variable or a volume mount.

Expose your sealed secret as an environment variable.

apiVersion: v1
kind: Pod
metadata:
  name: sealed-secret-pod
  annotations:
  #  io.katacontainers.config.hypervisor.cc_init_data: VALUE # Uncomment if init-data approach 
  #  io.katacontainers.config.hypervisor.kernel_params: VALUE # Uncomment if kernel params approach
spec:
  runtimeClassName: kata-qemu-coco-dev # use your CoCo runtime class
  containers:
    - name: busybox
      image: quay.io/prometheus/busybox:latest
      imagePullPolicy: Always
      command: ["/bin/sh", "-c", "echo $PROTECTED_SECRET"]
      env:
        - name: PROTECTED_SECRET
          valueFrom:
            secretKeyRef:
              name: sealed-secret
              key: secret

Expose your sealed secret as a volume mount.

apiVersion: v1
kind: Pod
metadata:
  name: sealed-secret-pod-volume
  annotations:
  #  io.katacontainers.config.hypervisor.cc_init_data: VALUE # Uncomment if init-data approach 
  #  io.katacontainers.config.hypervisor.kernel_params: VALUE # Uncomment if kernel params approach
spec:
  runtimeClassName: kata-qemu-coco-dev # use your CoCo runtime class
  containers:
    - name: busybox
      image: quay.io/prometheus/busybox:latest
      imagePullPolicy: Always
      command: ["/bin/sh", "-c", "cat /sealed/secret-value/secret"]
      volumeMounts:
        - name: sealed-secret-volume
          mountPath: "/sealed/secret-value"
  volumes:
    - name: sealed-secret-volume
      secret:
        secretName: sealed-secret

Note

Currently sealed secret volumes must be mounted in the /sealed directory.

Sealed secret formats

Sealed secrets are defined in two main formats: vault secrets and envelope secrets.

Vault Secrets

A vault secret is a pointer to a secret stored elsewhere, either in a KMS or a KBS. To fulfill a vault secret, the CDH retrieves the secret value from the selected provider during unsealing.

Creating a vault secret does not require encrypting the secret value itself. Instead, the sealed secret stores metadata that identifies where the plaintext secret can be retrieved.

The format of a vault sealed secret is:

{
  "version": "0.1.0",
  "type": "vault",
  "provider": "xxx",
  "name": "xxx",
  "provider_settings": {
    "...": "..."
  },
  "annotations": {
    "...": "..."
  }
}

Field descriptions:

version: REQUIRED format version. The current version is 0.1.0.
type: REQUIRED and must be vault.
provider: REQUIRED provider of the secret value.
name: REQUIRED identifier of the secret value used by the provider.
provider_settings: REQUIRED provider-specific configuration used to create the vault client.
annotations: optional provider-specific metadata used to retrieve the plaintext secret.

Envelope Secrets

You can also create envelope secrets. With envelope secrets, the secret value itself is included in the sealed secret, unlike a vault secret, which only contains a pointer to a secret stored elsewhere.

An envelope secret uses envelope encryption: a data encryption key encrypts the plaintext secret, and a KMS or KBS unwraps that key during unsealing. This allows the key used for unwrapping to remain in the KMS, while the sealed secret can still travel with the workload. It also decouples the protected secret material from Trustee/KBS resource storage.

The envelope model can be summarized as:

$$Sealed\ Secret := \{Enc_{Sealing\ key}(Encryption\ Key),\ Enc_{Encryption\ Key}(secret\ value)\}$$

The format of an envelope sealed secret is:

{
  "version": "0.1.0",
  "type": "envelope",
  "provider": "xxx",
  "key_id": "xxx",
  "encrypted_key": "ab27dc=",
  "encrypted_data": "xxx",
  "wrap_type": "A256GCM",
  "iv": "xxx",
  "provider_settings": {
    "...": "..."
  },
  "annotations": {
    "...": "..."
  }
}

Field descriptions:

version: REQUIRED format version. The current version is 0.1.0.
type: REQUIRED and must be envelope.
provider: REQUIRED provider of the sealing key.
key_id: REQUIRED identifier of the sealing key used to unwrap the data encryption key.
encrypted_key: REQUIRED wrapped data encryption key, base64 encoded.
encrypted_data: REQUIRED encrypted secret value, base64 encoded.
wrap_type: REQUIRED algorithm used by the data encryption key to encrypt the secret value. A256GCM is preferred.
iv: REQUIRED initialization vector used during secret encryption, base64 encoded.
provider_settings: REQUIRED provider-specific configuration used to create the KMS client.
annotations: optional provider-specific metadata used during unsealing.

Supported providers

Provider	Usage	Notes
`kbs`	Vault secrets	Built into the CoCo/Trustee flow
`aliyun`	Envelope secrets	See the Aliyun KMS guide
`ehsm`	Envelope secrets	See the eHSM guide

Integrity protection and signing

Sealed secrets support integrity protection with JWS. In practice the CLI emits a string that starts with sealed. followed by the protected header, payload, and signature as dot-separated base64url sections.

sealed.<base64url(JWS protected header)>.<base64url(JWS payload)>.<base64url(JWS signature)>

The JWS kid identifies the public key used to verify the signature. That public key can be:

provisioned directly as a CDH credential, or
stored in Trustee and referenced through a resource URI.

Using a resource URI for kid is convenient because the guest can fetch the verification key from Trustee during unsealing.

7 - Init-Data

Use Init-Data to inject dynamic configurations for Pods

Confidential Containers need to access configuration data that cannot be practically embedded in the OS image, such as:

URIs and certificates required to access Key Broker Services
Agent Policy that is supposed to be enforced by the policy engine
Configuration regarding image pull, such as proxies and configuration URIs.

In CoCo project this data is referred to as Init-Data. It is specified in TOML format.

Remote attestation ensures the integrity of Init-Data.

Agent Policy Overview

The Agent Policy is one of the most critical components you’ll configure via Init-Data. It acts as a security policy engine inside the TEE, controlling what operations the Kata agent can perform.

Why Agent Policies Matter

Confidential Containers run inside a TEE, but they still interact with the Kubernetes control plane. A malicious or compromised control plane could attempt to:

Execute arbitrary commands in your container (kubectl exec)
Launch unauthorized container images
Extract secrets or sensitive data

The agent policy enforces restrictions on these operations inside the TEE, preventing the control plane from compromising your workload’s integrity. More specifically, the agent policy restricts the API between the kata shim running in the host and the kata agent running in the confidential guest.

What Agent Policies Control

Agent policies are written in Rego and can control:

API Call	What It Allows	Example Use Case
`CreateContainerRequest`	Which container images can be launched	Allowlist specific image digests
`ExecProcessRequest`	Which commands can be executed	Block `kubectl exec` or allow specific commands only
`CopyFileRequest`	File operations	Restrict file copying to/from containers
`PullImageRequest`	Image pulling operations	Control which registries are accessible

See the complete Kata Agent API for all available request types.

Example: Restrictive Agent Policy

Here’s a policy that only allows specific images and blocks all exec operations:

package agent_policy

# Allow management operations
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GuestDetailsRequest := true
default StartContainerRequest := true
default StatsContainerRequest := true

# Block exec by default
default ExecProcessRequest := false

# Block container creation by default
default CreateContainerRequest := false

# Only allow specific image digests
CreateContainerRequest if {
	some storage in input.storages
	storage.source == "docker.io/library/nginx@sha256:e56797eab4a5300158cc015296229e13a390f82bfc88803f45b08912fd5e3348"
}

# Optionally allow specific commands
ExecProcessRequest if {
	input_command := concat(" ", input.process.Args)
	input_command == "whoami"
}

Generating Agent Policies with genpolicy

Writing agent policies by hand can be tedious and error-prone for complex workloads. The genpolicy tool automatically generates appropriate policies from your Kubernetes manifests.

Installing genpolicy

The genpolicy tool is included in Kata Containers releases:

Download from Kata Containers releases
Or build from source in the kata-containers repository

Using genpolicy to Create Policy for Init-Data

Step 1: Create your Kubernetes manifest

You can either write a pod manifest manually or generate one using kubectl:

# Option A: Create a pod.yaml manually
cat <<EOF > pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-secure
spec:
  runtimeClassName: kata-qemu-tdx
  containers:
  - name: nginx
    image: docker.io/library/nginx@sha256:e56797eab4a5300158cc015296229e13a390f82bfc88803f45b08912fd5e3348
EOF

# Option B: Generate using kubectl --dry-run
kubectl create deployment nginx-secure \
    --image=docker.io/library/nginx@sha256:e56797eab4a5300158cc015296229e13a390f82bfc88803f45b08912fd5e3348 \
    --dry-run=client \
    -o yaml > deployment.yaml

Step 2: Generate the policy

Run genpolicy on your manifest (works with YAML or JSON):

genpolicy -y pod.yaml > policy.rego
# or
genpolicy -y deployment.yaml > policy.rego

The generated policy will:

Allow the specific image digests referenced in your manifest
Set appropriate defaults for container operations
Include necessary management operations

Step 3: Review and customize

The generated policy is a starting point. Review it and customize based on your security requirements.

Step 4: Include in Init-Data TOML

Copy the contents of your policy file into your initdata.toml:

version = "0.1.0"
algorithm = "sha384"

[data]
"policy.rego" = '''
<paste contents of policy.rego here>
'''

"aa.toml" = '''
[token_configs.kbs]
url = "http://your-kbs:8080"
'''

"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "http://your-kbs:8080"
'''

Then follow the steps in the example below to embed the Init-Data in your pod.

For more details, see the genpolicy documentation.

Important

Always review generated policies before production use. Genpolicy creates reasonable defaults, but your specific security requirements may need additional customization.

Init-Data Format

Now that you understand agent policies, let’s see how to package them along with other configuration into Init-Data.

A typical Init-Data TOML file looks like this:

version = "0.1.0"
algorithm = "sha384"
[data]
"policy.rego" = '''
  <contents of agent-policy>
'''
"aa.toml" = '''
  <contents of Attestation-Agent configuration TOML>
'''

"cdh.toml" = '''
  <contents of Confidential Data Hub configuration TOML>
'''

where,

version: The format version of this Init-Data TOML. Currently, ONLY version 0.1.0 is supported.
algorithm: The hash algorithm used to calculate the digest of the Init-Data TOML. This is crucial for attestation. Acceptable values are sha256, sha384 and sha512.
data: A dictionary containing concrete configurable files. Supported entries include:
- "policy.rego": Kata Agent Policy file. This helps to restrict the pod to an expected state during runtime..
- "aa.toml": Configuration file for the Attestation Agent, responsible for remote attestation.
- "cdh.toml": Configuration file for the Confidential Data Hub, responsible for in-pod confidential data flow, including image pulling, image decryption, etc.

The detailed Init-Data format definition can be found in the referenced specification.

Example: Embedding Init-Data in a Pod

This section provides an example of creating Init-Data with an agent policy and deploying it to enable remote attestation.

Step 1: Prepare an Init-Data TOML

Suppose we have a KBS service listening at http://1.2.3.4:8080.

The example initdata.toml is as follows. Note that we need to set the KBS URL in the following sections:

url of [token_configs.kbs] section in "aa.toml", defining the KBS for remote attestation.
url of [kbc] section in "cdh.toml", defining the KBS for accessing confidential resources. In this case, we use the same KBS for both attestation and resource provision.

version = "0.1.0"
algorithm = "sha384"
[data]
"policy.rego" = '''
package agent_policy

# Allow management operations
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GuestDetailsRequest := true
default StartContainerRequest := true
default StatsContainerRequest := true

# Block exec by default
default ExecProcessRequest := false

# Block container creation by default
default CreateContainerRequest := false

# Only allow specific image digests
CreateContainerRequest if {
    some storage in input.storages
    storage.source == "docker.io/library/nginx@sha256:e56797eab4a5300158cc015296229e13a390f82bfc88803f45b08912fd5e3348"
}

# Optionally allow specific commands
ExecProcessRequest if {
    input_command := concat(" ", input.process.Args)
    input_command == "whoami"
}
'''
"aa.toml" = '''
[token_configs]
[token_configs.kbs]
url = "http://1.2.3.4:8080"
'''

"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "http://1.2.3.4:8080"
'''

Tip

The policy shown above restricts containers to a specific nginx image digest. For your workloads, use genpolicy to automatically generate appropriate policies as shown in the genpolicy section above.

Step 2: Embed the Init-Data in Pod YAML

We can gzip compress and base64 encode the Init-Data TOML.

initdata=$(cat initdata.toml | gzip | base64 -w0)

Then embed it in pod.yaml

cat <<EOF > pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: nginx-secure
  annotations:
    io.katacontainers.config.hypervisor.cc_init_data: ${initdata}
spec:
  runtimeClassName: kata-qemu-tdx
  containers:
    - name: nginx
      image: docker.io/library/nginx@sha256:e56797eab4a5300158cc015296229e13a390f82bfc88803f45b08912fd5e3348
      imagePullPolicy: Always
EOF

Once deployed, the pod will perform attestation with http://1.2.3.4:8080 whenever confidential resources need to be accessed.

kubectl apply -f pod.yaml

Integrity and Attestation

One of the key properties of Init-Data is that its integrity is cryptographically verified through remote attestation.

When you specify Init-Data:

A hash of the Init-Data is calculated using the algorithm specified in the TOML
This hash is embedded into the TEE’s attestation evidence:
- For TDX/SNP: Written to a configuration register at pod launch (TDX’s MR_CONFIG_ID, SNP’s HOSTDATA)
- For vTPM platforms: Extended into PCR 8 by the Attestation Agent after launch
For TDX/SNP, the Attestation Agent validates that the Init-Data hash matches the launch configuration (if mismatch, the pod fails to start)
During remote attestation, the Attestation Service verifies the hash matches the hardware evidence and includes the Init-Data claims in the attestation token
KBS resource policies can check these verified Init-Data claims in the token to enforce additional requirements before releasing secrets

This ensures that:

The agent policy hasn’t been tampered with
KBS/AA configuration is authentic
The workload is running with the intended security configuration
Resource policies can make access decisions based on verified configuration

For more details on the integrity mechanisms, see the Policing a Sandbox blog post.

Additional Use Cases

The dynamic configuration mechanism provided by Init-Data is highly flexible. Here are some more features that leverage Init-Data:

Local registries - Configure pods to pull from local container registries
Image pull proxy - Route image pulls through a proxy server

8 - Image Pull Proxy

Pull containers from self-hosted registries

In today’s cloud-native environments, directly pulling container images from public registries can introduce security and performance challenges. To mitigate these issues, an image pull proxy can act as an intermediary between your environment and external image registries. This document provides a detailed method for configuring a proxy server using Init-Data for CoCo.

Configuring an Image Pull Proxy with Initdata

It is easy to configure the image pull proxy via Init-Data.

version = "0.1.0"
algorithm = "sha256"
[data]
"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SetPolicyRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := true
'''
"aa.toml" = '''
[token_configs]
[token_configs.kbs]
url = "http://1.2.3.4:8080"
'''

"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "http://1.2.3.4:8080"

[image.image_pull_proxy]

# HTTPS proxy that will be used to pull image
#
# By default this value is not set.
https_proxy = "http://127.0.0.1:5432"

# HTTP proxy that will be used to pull image
#
# By default this value is not set.
http_proxy = "http://127.0.0.1:5432"

# No proxy env that will be used to pull image.
#
# This will ensure that when we access the image registry with specified
# IPs, both `https_proxy` and `http_proxy` will not be used.
#
# If neither `https_proxy` nor `http_proxy` is not set, this field will do nothing.
#
# By default this value is not set.
no_proxy = "192.168.0.1,localhost"
'''

Note: a new section [image.image_pull_proxy] has been added to "cdh.toml", and the field https_proxy, https_proxy and no_proxy are used to configure proxy-related settings in your environment.

By configuring these settings, you can control the proxy behavior for pulling images, ensuring both security and efficiency in your container deployments. Adjust the proxy addresses and no-proxy settings as needed to fit your network configuration.

9 - Local Registries

Pull containers from self-hosted registries

In certain scenarios, there arises a need to pull container images directly from local registries, rather than relying on public repositories. This requirement may stem from the desire for reduced latency, enhanced security, or network isolation.

Local registries can be categorized into two types based on their security protocols: HTTPS-secured registries and HTTP-accessible registries.

HTTPS-Secured Registries

For registries protected by HTTPS, communication security is paramount to ensure that image transmission remains confidential and is received intact. Registry certificate configuration needs to be configured via Init-Data for secure communication.

version = "0.1.0"
algorithm = "sha256"
[data]
"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SetPolicyRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := true
'''
"aa.toml" = '''
[token_configs]
[token_configs.kbs]
url = "http://1.2.3.4:8080"
'''

"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "http://1.2.3.4:8080"

[image]
# To support registries with self signed certs. This config item
# is used to add extra trusted root certifications. The certificates
# must be encoded by PEM.
#
# By default this value is not set.
extra_root_certificates = ["""
-----BEGIN CERTIFICATE-----
MIIFTDCCAvugAwIBAgIBADBGBgkqhkiG9w0BAQowOaAPMA0GCWCGSAFlAwQCAgUA
oRwwGgYJKoZIhvcNAQEIMA0GCWCGSAFlAwQCAgUAogMCATCjAwIBATB7MRQwEgYD
VQQLDAtFbmdpbmVlcmluZzELMAkGA1UEBhMCVVMxFDASBgNVBAcMC1NhbnRhIENs
YXJhMQswCQYDVQQIDAJDQTEfMB0GA1UECgwWQWR2YW5jZWQgTWljcm8gRGV2aWNl
...
-----END CERTIFICATE-----
"""]
'''

Note: a new section [image] has been added to "cdh.toml", and the field extra_root_certificates includes the certificate chain of the local registry.

HTTP-Accessible Registries

TODO

10 - Protected Storage

Options for confidential storage

By default, CoCo workloads execute in confidential guest memory. Files written to the workload filesystem will be stored in guest memory and protected by confidential computing.

Of course, cloud native workloads can leverage a wide variety of external storage options. Some of these might break the confidential trust model. Some can be used with adaptations to the workload (e.g. wrapping secrets before storing them). Carefully consider the trust model of any external storage volumes, services, or paradigms before attaching them to a confidential workload.

To simplify things, Confidential Containers provides some confidential storage primitives.

10.1 - Confidential EmptyDir

Protected ephemeral storage for workloads

When using a confidential runtime class, all emptyDir volumes will automatically be created on top of secure block devices. These confidential emptyDir volumes use LUKS2 on top of a block device provided by the host.

Confidential emptyDir can be a good fit for a workload that needs to write a lot of secret data to a scratch directory. If stored inside the guest, this data could deplete guest memory. Instead, the confidential emptyDir is backed by a block device provided by the host. The block device is encrypted inside the guest such that the host cannot access the data.

Warning

While confidential emptyDir volumes have integrity protection via LUKS2, they do not have replay protection. While the host cannot modify individual blocks, they can rollback the entire volume to some earlier state.

A confidential emptyDir can be added to a workload the same way a traditional emptyDir would be used.

volumeMounts:
      - name: scratch-volume 
        mountPath: /scratch-directory
  volumes:
  - name: scratch-volume
    emptyDir:
      sizeLimit: 64Gi

On the host, this volume will be backed by a sparse file. As such, host resource usage will initially be small.

Note

LUKS2 does not support discard/trim when integrity is enabled. Thus, when a file is removed from the emptyDir by the guest, the size of the sparse file will not decrease. Over the lifespan of the workload, the size of the sparse file can only increase.

Confidential emptyDir volumes are ephemeral. They are removed when the pod is torn down.

The LUKS2 header for the volume is stored in guest memory and is not accessible to the host.

If you want to use an emptyDir that isn’t backed by a LUKS volume, set the emptyDir medium to Memory. This will create an emptyDir that is stored in guest memory.

volumeMounts:
  - name: memory-empty-vol
    mountPath: "/tmp/cache"
volumes:
  - name: memory-empty-vol
  emptyDir:
    medium: Memory
    sizeLimit: "50M"

11 - Runtime Attestation

Measurement from workload at runtime

Workloads can request runtime attestation of arbitrary data via a generic interface. Not all hardware platforms support runtime attestation, but those that do will fulfill the request.

On these platforms, the Attestation Agent maintains an event log, which tracks attestation events. This log will be forwarded to Trustee via the KBS protocol and compared to the hardware evidence.

Enabling

To enable this feature, set the following parameter in the guest kernel command line.

agent.guest_components_rest_api=all

Warning

Note that this configuration will also allow the workload to fetch attestation reports at runtime. In some configurations this can be dangerous. See attestation report page for more information.

The Attestation Agent configuration must also have runtime attestation enabled. This can be set via Init-Data, with the [eventlog_config] section below. This configuration can also specify the measurement index that will be used. On platforms with limited indices, the index will be mapped to the available registers. For example, on TDX setting the measurement index to 17 will usually result in extending RTMR 3.

version = "0.1.0"
algorithm = "sha384"
[data]
"aa.toml" = '''
[token_configs]
[token_configs.kbs]
url = "http://<trustee-uri>"

[eventlog_config]
init_pcr = 17
enable_eventlog = true
'''

"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "http://<trustee-uri>"
'''

The above configurations can be added to a workload as annotations. See the Init-Data page for more information on using Init-Data.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        io.katacontainers.config.hypervisor.kernel_params: "agent.guest_components_rest_api=all"
        io.katacontainers.config.hypervisor.cc_init_data: "H4sIAAAAAAAAA4WOwQ6DIBBE7/sVhos3xGpSa9IvMYQgUiEiGFz9/kLTHtpLjzO7M29OHXcbfHEvCKM1ZQSkm0O0aNbs7UY2XUtgmCRKDkRKimF1JN3KsoQBw6K9UME/7LzzH02XMXlHdLnJIG59VdWMXtqWMnpr+o51iQeDPrVHF+Z3joP1FsWmYsrVV9Bejk6Lz1cyMR4aMh+Imsz3omVUHLxcdYYqJZImfze8up7xdiRhCwEAAA=="
    spec:
      runtimeClassName: # (...)
      containers:
      - name: nginx
        # (...)

Runtime Attestation

Once enabled, runtime attestation can be triggered via the rest API.

curl -X POST http://127.0.0.1:8006/aa/aael \
     -H "Content-Type: application/json" \
     -d '{"domain":"test","operation":"test","content":"test"}'

The domain and operation are context fields that will be included in the event log.

You can check that the event log was updated by inspecting an attestation token.

curl http://127.0.0.1:8006/aa/token\?token_type\=kbs | jq -r '.token |split(".") | .[1] | @base64d | fromjson'

The event log may contain boot-time entries, but at the end you should see your entry.

{
   "details":{
      "data":{
         "content":"test",
         "domain":"test",
         "operation":"test"
      },
      "string":"test test test",
      "unicode_name":"AAEL"
   },
   "digest_matches_event":true,
   "digests":[
      {
         "alg":"SHA-384",
         "digest":"1495be3eb2120e59facb8f92447d64f..."
      }
   ],
   "event":"TEVBQREAAAB0ZXN0IHRlc3QgdGVzenp6dA==",
   "index":4,
   "type_name":"EV_EVENT_TAG"
}

Features

1 - Get Attestation

Enabling

Attestation

Security

2 - Get Secret Resources

3 - Signed Images

Overview

Creating an Image

Creating a Key Pair

Signing the Image

cosign v3.0.x

cosign >= v2.2.0 and < v3.0

Running an Image

Setting the Security Policy for Signed Images

Enable Signature Verification

Run a Signed Workload

Troubleshooting

See Also

Cosign GitHub Integration

4 - Encrypted Images

Context

Instructions

Encrypt an image

Provision image key

Launch a Pod

Debugging

0. Launching with an unencrypted image

1. Retrieve hardware evidence from TEE

2. Perform remote attestation

3. Retrieve decryption key from KBS

4. Other (desperate) measures

5 - Authenticated Registries

Context

Instructions

Create registry authentication file

Provision the registry authentication file

Launch a Pod

6 - Sealed Secrets

Basic Usage

Creating a sealed secret

Adding a sealed secret to Kubernetes

Configuring the KBS for your workload

Upload the secret value to the KBS

Upload the JWK public key to the KBS

Deploying a sealed secret to a confidential workload

Sealed secret formats

Vault Secrets

Envelope Secrets

Supported providers

Integrity protection and signing

7 - Init-Data

Agent Policy Overview

Why Agent Policies Matter

What Agent Policies Control

Example: Restrictive Agent Policy

Generating Agent Policies with genpolicy

Installing genpolicy

Using genpolicy to Create Policy for Init-Data

Init-Data Format

Example: Embedding Init-Data in a Pod

Step 1: Prepare an Init-Data TOML

Step 2: Embed the Init-Data in Pod YAML

Integrity and Attestation

Additional Use Cases

8 - Image Pull Proxy

Configuring an Image Pull Proxy with Initdata

9 - Local Registries

HTTPS-Secured Registries

HTTP-Accessible Registries

10 - Protected Storage

10.1 - Confidential EmptyDir

11 - Runtime Attestation

Enabling

Runtime Attestation

cosign `v3.0.x`

cosign `>= v2.2.0` and `< v3.0`