Features

• 1: Authenticated Registries
• 2: Encrypted Images
• 3: Local Registries
• 4: Protected Storage
• 5: Sealed Secrets
• 6: Signed Images

1 - Authenticated Registries

Context

A user might want to use container images from private OCI registries, hence requiring authentication. The project provides the means to pull protected images from authenticated registry.

Important: ideally the authentication credentials should only be accessible from within a Trusted Execution Environment, however, due some limitations on the architecture of components used by CoCo, the credentials need to be exposed to the host, thus registry authentication is not currently a confidential feature. The community has worked to remediate that limitation and, in meanwhile, we recommend the use of encrypted images as a mitigation.

Instructions

The following steps require a functional CoCo installation on a Kubernetes cluster. A Key Broker Client (KBC) has to be configured for TEEs to be able to retrieve confidential secrets. We assume cc_kbc as a KBC for the CoCo project’s Key Broker Service (KBS) in the following instructions, but authenticated registries should work with other Key Broker implementations in a similar fashion.

Create registry authentication file

The registry authentication file should have the containers-auth.json format, with exception of credential helpers (credHelpers) that aren’t supported. Also it’s not supported glob URLs nor prefix-matched paths as in Kubernetes interpretation of config.json.

Create the registry authentication file (e.g containers-auth.json) like this:

export AUTHENTICATED_IMAGE="my-registry.local/repository/image:latest"
export AUTHENTICATED_IMAGE_NAMESPACE="$(echo "$AUTHENTICATED_IMAGE" | cut -d':' -f1)"
export AUTHENTICATED_IMAGE_USER="MyRegistryUser"
export AUTHENTICATED_IMAGE_PASSWORD="MyRegistryPassword"
cat <<EOF>> containers-auth.json
{
	"auths": {
		"${AUTHENTICATED_IMAGE_NAMESPACE}": {
			"auth": "$(echo ${AUTHENTICATED_IMAGE_USER}:${AUTHENTICATED_IMAGE_PASSWORD} | base64 -w 0)"
		}
	}
}
EOF

Where:

• AUTHENTICATED_IMAGE is the full-qualified image name
• AUTHENTICATED_IMAGE_NAMESPACE is the image name without the tag
• AUTHENTICATED_IMAGE_USER and AUTHENTICATED_IMAGE_PASSWORD are the registry credentials user and password, respectively
• auth’s value is the colon-separated user and password (user:password) credentials string encoded in base64

Provision the registry authentication file

Prior to launching a Pod the registry authentication file needs to be provisioned to the Key Broker’s repository. For a KBS deployment on Kubernetes using the local filesystem as repository storage it would work like this:

export KEY_PATH="default/containers/auth"
kubectl exec deploy/kbs -c kbs -n coco-tenant -- mkdir -p "/opt/confidential-containers/kbs/repository/$(dirname "$KEY_PATH")"
cat containers-auth.json | kubectl exec -i deploy/kbs -c kbs -n coco-tenant -- tee "/opt/confidential-containers/kbs/repository/${KEY_PATH}" > /dev/null

The CoCo infrastructure components need to cooperate with containerd and nydus-snapshotter to pull the container image from TEE. Currently the nydus-snapshotter needs to fetch the image’s metadata from registry, then authentication credentials are read from a Kubernetes secret of docker-registry type. So it should be created a secret like this:

export SECRET_NAME="cococred"
kubectl create secret docker-registry "${SECRET_NAME}" --docker-server="https://${AUTHENTICATED_IMAGE_NAMESPACE}" \
    --docker-username="${AUTHENTICATED_IMAGE_USER}" --docker-password="${AUTHENTICATED_IMAGE_PASSWORD}"

Where:

• SECRET_NAME is any secret name

Launch a Pod

Create the pod yaml (e.g. pod-image-auth.yaml) like below and apply it:

export KBS_ADDRESS="172.18.0.3:31731"
export RUNTIMECLASS="kata-qemu-coco-dev"
cat <<EOF>> pod-image-auth.yaml
apiVersion: v1
kind: Pod
metadata:
  name: image-auth-feat
  annotations:
    io.containerd.cri.runtime-handler: ${RUNTIMECLASS}
    io.katacontainers.config.hypervisor.kernel_params: ' agent.image_registry_auth=kbs:///${KEY_PATH} agent.guest_components_rest_api=resource agent.aa_kbc_params=cc_kbc::http://${KBS_ADDRESS}'
spec:
  runtimeClassName: ${RUNTIMECLASS}
  containers:
    - name: test-container
      image: ${AUTHENTICATED_IMAGE}
      imagePullPolicy: Always
      command:
        - sleep
        - infinity
  imagePullSecrets:
    - name: ${SECRET_NAME}
EOF

Where:

• KBS_ADDRESS is the host:port address of KBS
• RUNTIMECLASS is any of available CoCo runtimeclassess (e.g. kata-qemu-tdx, kata-qemu-snp). For this example, kata-qemu-coco-dev allows to create CoCo pod on systems without confidential hardware. It should be replaced with a class matching the TEE in use.

What distinguish the pod specification for authenticated registry from a regular CoCo pod is:

• the agent.image_registry_auth property in io.katacontainers.config.hypervisor.kernel_params annotation indicates the location of the registry authentication file as a resource in the KBS
• the imagePullSecrets as required by nydus-snapshotter

Check the pod gets Running:

kubectl get -f pod-image-auth.yaml
NAME              READY   STATUS    RESTARTS   AGE
image-auth-feat   1/1     Running   0          2m52s

2 - Encrypted Images

Context

A user might want to bundle sensitive data on an OCI (Docker) image. The image layers should only be accessible within a Trusted Execution Environment (TEE).

The project provides the means to encrypt an image with a symmetric key that is released to the TEE only after successful verification and appraisal in a Remote Attestation process. CoCo infrastructure components within the TEE will transparently decrypt the image layers as they are pulled from a registry without exposing the decrypted data outside the boundaries of the TEE.

Instructions

The following steps require a functional CoCo installation on a Kubernetes cluster. A Key Broker Client (KBC) has to be configured for TEEs to be able to retrieve confidential secrets. We assume cc_kbc as a KBC for the CoCo project’s Key Broker Service (KBS) in the following instructions, but image encryption should work with other Key Broker implementations in a similar fashion.

Encrypt an image

We extend public image with secret data.

docker build -t unencrypted - <<EOF
FROM nginx:stable
RUN echo "something confidential" > /secret
EOF

The encryption key needs to be a 32 byte sequence and provided to the encryption step as base64-encoded string.

KEY_FILE="image_key"
head -c 32 /dev/urandom | openssl enc > "$KEY_FILE"
KEY_B64="$(base64 < $KEY_FILE)"

The key id is a generic resource descriptor used by the key broker to look up secrets in its storage. For KBS this is composed of three segments: $repository_name/$resource_type/$resource_tag

KEY_PATH="/default/image_key/nginx"
KEY_ID="kbs://${KEY_PATH}"

The image encryption logic is bundled and invoked in a container:

git clone https://github.com/confidential-containers/guest-components.git
cd guest-components
docker build -t coco-keyprovider -f ./attestation-agent/docker/Dockerfile.keyprovider .

To access the image from within the container, Skopeo can be used to buffer the image in a directory, which is then made available to the container. Similarly, the resulting encrypted image will be put into an output directory.

mkdir -p oci/{input,output}
skopeo copy docker-daemon:unencrypted:latest dir:./oci/input
docker run -v "${PWD}/oci:/oci" coco-keyprovider /encrypt.sh -k "$KEY_B64" -i "$KEY_ID" -s dir:/oci/input -d dir:/oci/output

We can inspect layer annotations to confirm the expected encryption was applied:

skopeo inspect dir:./oci/output | jq '.LayersData[0].Annotations["org.opencontainers.image.enc.keys.provider.attestation-agent"] | @base64d | fromjson'

Sample output:

{
  "kid": "kbs:///default/image_key/nginx",
  "wrapped_data": "lGaLf2Ge5bwYXHO2g2riJRXyr5a2zrhiXLQnOzZ1LKEQ4ePyE8bWi1GswfBNFkZdd2Abvbvn17XzpOoQETmYPqde0oaYAqVTMcnzTlgdYYzpWZcb3X0ymf9bS0gmMkqO3dPH+Jf4axXuic+ITOKy7MfSVGTLzay6jH/PnSc5TJ2WuUJY2rRtNaTY65kKF2K9YP6mtYBqcHqvPDlFiVNNeTAGv2w1zwaMlgZaSHV+Z1y+xxbOV5e98bxuo6861rMchjCiE7FY37PHD3a5ISogq90=",
  "iv": "Z8bGQL7r6qxSpd4L",
  "wrap_type": "A256GCM"
}

Finally, the resulting encrypted image can be provisioned to an image registry.

ENCRYPTED_IMAGE=some-private.registry.io/coco/nginx:encrypted
skopeo copy dir:./oci/output "docker://${ENCRYPTED_IMAGE}"

Provision image key

Prior to launching a Pod the image key needs to be provisioned to the Key Broker’s repository. For a KBS deployment on Kubernetes using the local filesystem as repository storage it would work like this:

kubectl exec deploy/kbs -- mkdir -p "/opt/confidential-containers/kbs/repository/$(dirname "$KEY_PATH")"
cat "$KEY_FILE" | kubectl exec -i deploy/kbs -- tee "/opt/confidential-containers/kbs/repository/${KEY_PATH}" > /dev/null

Note: If you’re not using KBS deployment using trustee operator additional namespace may be needed -n coco-tenant.

Launch a Pod

We create a simple deployment using our encrypted image. As the image is being pulled and the CoCo components in the TEE encounter the layer annotations that we saw above, the image key will be retrieved from the Key Broker using the annotated Key ID and the layers will be decrypted transparently and the container should come up.

In this example we default to the Cloud API Adaptor runtime, adjust this depending on the CoCo installation.

kubectl get runtimeclass -o jsonpath='{.items[].handler}'

Sample output:

kata-remote

Export variable:

CC_RUNTIMECLASS=kata-remote

Export KBS address:

KBS_ADDRESS=scheme://host:port

Deploy sample pod:

• Bare metal
• Cloud Api Adaptor

cat <<EOF> nginx-encrypted.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-encrypted
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        io.katacontainers.config.hypervisor.kernel_params: "agent.aa_kbc_params=cc_kbc::${KBS_ADDRESS}"
        io.containerd.cri.runtime-handler: ${CC_RUNTIMECLASS}
    spec:
      runtimeClassName: ${CC_RUNTIMECLASS}
      containers:
      - image: ${ENCRYPTED_IMAGE}
        name: nginx
        imagePullPolicy: Always
EOF
kubectl apply -f nginx-encrypted.yaml

We can confirm that the image key has been retrieved from KBS.

kubectl logs -f deploy/kbs | grep "$KEY_PATH"
[2024-01-23T10:24:52Z INFO  actix_web::middleware::logger] 10.244.0.1 "GET /kbs/v0/resource/default/image_key/nginx HTTP/1.1" 200 530 "-" "attestation-agent-kbs-client/0.1.0" 0.000670

Note: If you’re not using KBS deployment using trustee operator additional namespace may be needed -n coco-tenant.

3 - Local Registries

TODO

4 - Protected Storage

TODO

5 - Sealed Secrets

Sealed secrets allow confidential information to be stored in the untrusted control plane. Like normal Kubernetes secrets, sealed secrets are orchestrated by the control plane and are transparently provisioned to your workload as environment variables or volumes.

Basic Usage

Here’s how you create a vault secret. There are also envelope secrets, which are described later. Vault secrets are a pointer to resource stored in a KBS, while envelope secrets are wrapped secrets that are unwrapped with a KMS.

Creating a sealed secret

There is a helper tool for sealed secrets in the Guest Components repository.

Clone the repository.

git clone https://github.com/confidential-containers/guest-components.git

Inside the guest-components directory, you can build and run the tool with Cargo.

cargo run -p confidential-data-hub --bin secret

With the tool you can create a secret.

cargo run -p confidential-data-hub --bin secret seal vault --resource-uri kbs:///your/secret/here --provider kbs

A vault secret is fulfilled by retrieving a secret from a KBS inside the guest. The locator of your secret is specified by resource-uri.

This command should return a base64 string which you will use in the next step.

Adding a sealed secret to Kubernetes

Create a secret from your secret string using kubectl.

kubectl create secret generic sealed-secret --from-literal='secret=sealed.fakejwsheader.ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvc2VhbGVkLXNlY3JldC90ZXN0IiwKICAgICJwcm92aWRlciI6ICJrYnMiLAogICAgInByb3ZpZGVyX3NldHRpbmdzIjoge30sCiAgICAiYW5ub3RhdGlvbnMiOiB7fQp9Cg==.fakesignature'

When using --from-literal you provide a mapping of secret keys and values. The secret value should be the string generated in the previous step. The secret key can be whatever you want, but make sure to use the same one in future steps. This is separate from the name of the secret.

Deploying a sealed secret to a confidential workload

You can add your sealed secret to a workload yaml file.

You can expose your sealed secret as an environment variable.

apiVersion: v1
kind: Pod
metadata:
  name: sealed-secret-pod
spec:
  runtimeClassName: kata-qemu-coco-dev
  containers:
  - name: busybox
    image: quay.io/prometheus/busybox:latest
    imagePullPolicy: Always
    command: ["echo", "$PROTECTED_SECRET"]
    env:
    - name: PROTECTED_SECRET
      valueFrom:
        secretKeyRef:
          name: sealed-secret
          key: secret

You can also expose your secret as a volume.

apiVersion: v1
kind: Pod
metadata:
  name: secret-test-pod-cc
spec:
  runtimeClassName: kata
  containers:
  - name: busybox
    image: quay.io/prometheus/busybox:latest
    imagePullPolicy: Always
    command: ["cat", "/sealed/secret-value/secret"]
    volumeMounts:
        - name: sealed-secret-volume
          mountPath: "/sealed/secret-value"
  volumes:
    - name: sealed-secret-volume
      secret:
        secretName: sealed-secret

Advanced

Envelope Secrets

You can also create envelope secrets. With envelope secrets, the secret itself is included in the secret (unlike a vault secret, which is just a pointer to a secret). In an envelope secret, the secret value is wrapped and can be unwrapped by a KMS. This allows us to support models where the key for unwrapping secrets never leaves the KMS. It also decouples the secret from the KBS.

We currently support two KMSes for envelope secrets. See specific instructions for aliyun kms and eHSM.

6 - Signed Images

TODO