This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Features

Primitives provided by Confidential Containers

1: Get Attestation
2: Get Secret Resources
3: Signed Images
4: Encrypted Images
5: Authenticated Registries
6: Sealed Secrets
7: Init-Data
8: Image Pull Proxy
9: Local Registries
10: Protected Storage

In addition to running pods inside of enclaves, Confidential Containers provides several other features that can be used to protect workloads and data. Securing complex workloads often requires using some of these features.

Most features depend on and require attestation, which is described in the next section.

1 - Get Attestation

Workloads that request attestation evidence

Warning

This feature is disabled by default due to some subtle security considerations which are described below.

Workloads can directly request attestation evidence. A workload could use this evidence to carry out its own attestation protocol.

Enabling

To enable this feature, set the following parameter in the guest kernel command line.

agent.guest_components_rest_api=all

As usual, command line configurations can be added with annotations.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        io.katacontainers.config.hypervisor.kernel_params: "agent.guest_components_rest_api=all"
    spec:
      runtimeClassName: (...)
      containers:
      - name: nginx
        (...)

Attestation

Once enabled, an attestation can be retrieved via the REST API.

curl http://127.0.0.1:8006/aa/evidence?runtime_data=xxxx

The API expects runtime data to be provided. The runtime data will be included in the report. Typically this is used to bind a nonce or the hash of a key to the evidence.

Security

Enabling this feature can make your workload vulnerable to so-called “evidence factory” attacks. By default, different CoCo workloads can have the same TCB, even if they use different container images. This is a design feature, but it means that when the attestation report API is enabled, two workloads could produce interchangeable attestation reports, even if they are operated by different parties.

To make sure someone else cannot generate evidence that could be used with your attestation protocol, configure your guest to differentiate its TCB. For example, use the init-data (which is measured) to specify the public key of the KBS. Init-data is currently only supported with peer pods. More information about it will be added soon.

2 - Get Secret Resources

Workloads that request resources from Trustee

Note

Requesting secret resources depends on attestation. Configure attestation before requesting resources

While sealed secrets can be used to store encrypted secret resources as Kubernetes Secrets and have them transparently decrypted using keys from Trustee, a workload can also explicitly request resources via a REST API that is automatically exposed to the pod network namespace.

For example, you can run this command from your container.

curl http://127.0.0.1:8006/cdh/resource/default/key/1

In this example Trustee will fulfill the request for kbs:///default/key/1 assuming that the resource has been provisioned.

3 - Signed Images

Procedures to generate and deploy signed OCI images with CoCo

Overview

Encrypted images provide confidentiality, but they do not provide authenticity or integrity. Image signatures provide this additional property, preventing certain types of image tampering, for example.

In this brief guide, we show two tools that can be used to sign container images: cosign and skopeo. The skopeo tool can be used to create both cosign signatures or “simple signatures” (which leverage gpg keys). For our purposes, our skopeo examples will use the simple signing approach. In any case, the general approach is to

Create keys for signing,
Sign a newly tagged image, and
Update the KBS with the public signature key and a security policy.

Creating an Image

Creating Keys

Create a keypair using one of two approaches, cosign or Simple Signing - gpg.

To generate a public-private keypair with cosign, provide your COSIGN_PASSWORD and use the generate-key-pair action:

$ COSIGN_PASSWORD=just1testing2password3 cosign generate-key-pair

This will create the private and public keys: cosign.key and cosign.pub.

skopeo depends on gpg for a keypair. To generate a keypair with gpg using default options, use --full-generate-key:

$ gpg --full-generate-key

There are several prompts. A user for test purposes could be:

Github Runner
git@runner.com
just1testing2password3

Then export it. The --export-secret-key option is sufficient for exporting both the secret and public key. Example command:

$ gpg --export-secret-key F63DB2A1AB7C7F195F698C9ED9582CADF7FBCC5D > github-runner.keys

The keys can later be imported by gpg in a CI system using --batch to avoid typing the password:

$ gpg --batch --import ./github-runner.keys

When automating CI or test workflows, you can place the password for the key in a plaintext file (when it is safe to do so):

echo just1testing2password3 > git-runner-password.txt

Signing the Image

Sign the image using one of two approaches, cosign or Simple Signing - skopeo.

Use the private key to sign an image. In this example, we assume that there is a Dockerfile (your_dockerfile below) for creating an image that you want signed. The workflow is to build the image, push it to ghcr (which requires docker login), and sign it.

COCO_PKG=confidential-containers/test-container
docker build \
  -t ghcr.io/$(COCO_PKG):cosign-sig \
  -f <your_dockerfile> \
  .
docker push ghcr.io/$(COCO_PKG):cosign-sig
cosign sign --key ./cosign.key ghcr.io/${COCO_PKG}:cosign-sig

Ensure you have a gpg key owned by the user signing the image. See the previous subsection for generating and importing gpg keys.

Sign the image. For example, the following command uses an insecure-policy to sign a local image called confidential-containers/test-container. It uses the unsigned tag, and in the process of signing it, creates a new simple-signed tag. In this example, the resulting image is pushed to ghcr, which requires docker login:

COCO_PKG=confidential-containers/test-container
skopeo \
  copy \
  --debug \
  --insecure-policy \
  --sign-by git@runner.com \
  --sign-passphrase-file ./git-runner-password.txt \
  docker-daemon:ghcr.io/${COCO_PKG}:unsiged \
  docker://ghcr.io/${COCO_PKG}:simple-signed

Running an Image

Running a workload with a signed image is very similar to running workloads with unsigned images. The main difference is that for a signed image, you must provide some details to the KBS. This security policy tells the KBS which image is signed, the type of its key signature, and where to find the public key for verifying the signature. Beyond this, you run the workload as you usually would (e.g. via kubectl apply).

Setting the Security Policy for Signed Images

mkdir -p ${KBS_DIR_PATH}/data/kbs-storage/default/cosign-key \
  && cp cosign.pub ${KBS_DIR_PATH}/data/kbs-storage/default/cosign-key/1

Edit an image pulling validation policy file. Here is a sample policy file security-policy.json:

{
    "default": [{"type": "reject"}],
    "transports": {
        "docker": {
            "[REGISTRY_URL]": [
                {
                    "type": "sigstoreSigned",
                    "keyPath": "kbs:///default/cosign-key/1"
                }
            ]
        }
    }
}

Be sure to replace [REGISTRY_URL] with the desired registry URL of the encrypted image.

Lastly, register the image pulling validation policy file with KBS storage:

mkdir -p ${KBS_DIR_PATH}/data/kbs-storage/default/security-policy
cp security-policy.json ${KBS_DIR_PATH}/data/kbs-storage/default/security-policy/test

4 - Encrypted Images

Procedures to encrypt and consume OCI images in a TEE

Context

A user might want to bundle sensitive data on an OCI (Docker) image. The image layers should only be accessible within a Trusted Execution Environment (TEE).

The project provides the means to encrypt an image with a symmetric key that is released to the TEE only after successful verification and appraisal in a Remote Attestation process. CoCo infrastructure components within the TEE will transparently decrypt the image layers as they are pulled from a registry without exposing the decrypted data outside the boundaries of the TEE.

Instructions

The following steps require a functional CoCo installation on a Kubernetes cluster. A Key Broker Client (KBC) has to be configured for TEEs to be able to retrieve confidential secrets. We assume cc_kbc as a KBC for the CoCo project’s Key Broker Service (KBS) in the following instructions, but image encryption should work with other Key Broker implementations in a similar fashion.

Note

Please ensure you have a recent version of Skopeo (v1.13.3+) installed locally.

Encrypt an image

We extend public image with secret data.

docker build -t unencrypted - <<EOF
FROM nginx:stable
RUN echo "something confidential" > /secret
EOF

The encryption key needs to be a 32 byte sequence and provided to the encryption step as base64-encoded string.

KEY_FILE="image_key"
head -c 32 /dev/urandom | openssl enc > "$KEY_FILE"
KEY_B64="$(base64 < $KEY_FILE)"

The key id is a generic resource descriptor used by the key broker to look up secrets in its storage. For KBS this is composed of three segments: $repository_name/$resource_type/$resource_tag

KEY_PATH="/default/image_key/nginx"
KEY_ID="kbs://${KEY_PATH}"

The image encryption logic is bundled and invoked in a container:

git clone https://github.com/confidential-containers/guest-components.git
cd guest-components
docker build -t coco-keyprovider -f ./attestation-agent/docker/Dockerfile.keyprovider .

To access the image from within the container, Skopeo can be used to buffer the image in a directory, which is then made available to the container. Similarly, the resulting encrypted image will be put into an output directory.

mkdir -p oci/{input,output}
skopeo copy docker-daemon:unencrypted:latest dir:./oci/input
docker run -v "${PWD}/oci:/oci" coco-keyprovider /encrypt.sh -k "$KEY_B64" -i "$KEY_ID" -s dir:/oci/input -d dir:/oci/output

We can inspect layer annotations to confirm the expected encryption was applied:

skopeo inspect dir:./oci/output | jq '.LayersData[0].Annotations["org.opencontainers.image.enc.keys.provider.attestation-agent"] | @base64d | fromjson'

Sample output:

{
  "kid": "kbs:///default/image_key/nginx",
  "wrapped_data": "lGaLf2Ge5bwYXHO2g2riJRXyr5a2zrhiXLQnOzZ1LKEQ4ePyE8bWi1GswfBNFkZdd2Abvbvn17XzpOoQETmYPqde0oaYAqVTMcnzTlgdYYzpWZcb3X0ymf9bS0gmMkqO3dPH+Jf4axXuic+ITOKy7MfSVGTLzay6jH/PnSc5TJ2WuUJY2rRtNaTY65kKF2K9YP6mtYBqcHqvPDlFiVNNeTAGv2w1zwaMlgZaSHV+Z1y+xxbOV5e98bxuo6861rMchjCiE7FY37PHD3a5ISogq90=",
  "iv": "Z8bGQL7r6qxSpd4L",
  "wrap_type": "A256GCM"
}

Finally, the resulting encrypted image can be provisioned to an image registry.

ENCRYPTED_IMAGE=some-private.registry.io/coco/nginx:encrypted
skopeo copy dir:./oci/output "docker://${ENCRYPTED_IMAGE}"

Provision image key

Prior to launching a Pod the image key needs to be provisioned to the Key Broker’s repository. For a KBS deployment on Kubernetes using the local filesystem as repository storage it would work like this:

kubectl exec deploy/kbs -- mkdir -p "/opt/confidential-containers/kbs/repository/$(dirname "$KEY_PATH")"
cat "$KEY_FILE" | kubectl exec -i deploy/kbs -- tee "/opt/confidential-containers/kbs/repository/${KEY_PATH}" > /dev/null

Note: If you’re not using KBS deployment using trustee operator additional namespace may be needed -n coco-tenant.

Launch a Pod

We create a simple deployment using our encrypted image. As the image is being pulled and the CoCo components in the TEE encounter the layer annotations that we saw above, the image key will be retrieved from the Key Broker using the annotated Key ID and the layers will be decrypted transparently and the container should come up.

In this example we default to the Cloud API Adaptor runtime, adjust this depending on the CoCo installation.

kubectl get runtimeclass -o jsonpath='{.items[].handler}'

Sample output:

kata-remote

Export variable:

CC_RUNTIMECLASS=kata-remote

Export KBS address:

KBS_ADDRESS=scheme://host:port

Deploy sample pod:

cat <<EOF> nginx-encrypted.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-encrypted
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        io.katacontainers.config.hypervisor.kernel_params: "agent.aa_kbc_params=cc_kbc::${KBS_ADDRESS}"
        io.containerd.cri.runtime-handler: ${CC_RUNTIMECLASS}
    spec:
      runtimeClassName: ${CC_RUNTIMECLASS}
      containers:
      - image: ${ENCRYPTED_IMAGE}
        name: nginx
        imagePullPolicy: Always
EOF
kubectl apply -f nginx-encrypted.yaml

Create file $HOME/initdata.toml

cat <<EOF> initdata.toml
algorithm = "sha256"
version = "0.1.1"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]
url = '${KBS_ADDRESS}'

[token_configs.kbs]
url = '${KBS_ADDRESS}'
'''

"cdh.toml"  = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = 'cc_kbc'
url = '${KBS_ADDRESS}'
'''
EOF

Export variable:

INIT_DATA_B64=$(cat $HOME/initdata.toml | gzip | base64 -w0)

Deploy:

cat <<EOF> nginx-encrypted.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
  name: nginx-encrypted
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
      annotations:
        io.katacontainers.config.runtime.cc_init_data: "${INIT_DATA_B64}"
        io.containerd.cri.runtime-handler: ${CC_RUNTIMECLASS}
    spec:
      runtimeClassName: ${CC_RUNTIMECLASS}
      containers:
      - image: ${ENCRYPTED_IMAGE}
        name: nginx
        imagePullPolicy: Always
EOF
kubectl apply -f nginx-encrypted.yaml

We can confirm that the image key has been retrieved from KBS.

kubectl logs -f deploy/kbs | grep "$KEY_PATH"
[2024-01-23T10:24:52Z INFO  actix_web::middleware::logger] 10.244.0.1 "GET /kbs/v0/resource/default/image_key/nginx HTTP/1.1" 200 530 "-" "attestation-agent-kbs-client/0.1.0" 0.000670

Note: If you’re not using KBS deployment using trustee operator additional namespace may be needed -n coco-tenant.

Debugging

The encrypted image feature relies on a cascade of other features as building blocks:

Image decryption is built on secret retrieval from KBS
Secret retrieval is built on remote attestation
Remote attestation requires retrieval of hardware evidence from the TEE

All of the above have to work in order to decrypt an encrypted image.

0. Launching with an unencrypted image

Launch the same image with unencrypted layers from the same registry to verify that the image itself is not an issue.

1. Retrieve hardware evidence from TEE

Launch an unencrypted library/nginx deployment named nginx with a CoCo runtime class. Issue kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/aa/evidence\?runtime_data\=xxxx. This should produce a real hardware evidence rendered as JSON. Something like {"svn":"1","report_data":"eHh4eA=="} is not real hardware evidence, it’s a dummy response from the sample attester. Real TEE evidence should be more verbose and contain certificates and measurements.

The reason for not producing real evidence could be a wrong build of Attestation Agent for the TEE that you are attempting to use, or the Confidential VM not exposing the expected interfaces.

Note: In some configurations of the CoCo image the facility to retrieve evidence is disabled by default. For bare-metal CoCo images you can enable it by setting agent.guest_components_rest_api=all on the kernel cmdline (see here).

2. Perform remote attestation

Run kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/aa/token\?token_type\=kbs. This should produce an attestation token. If you don’t receive a token but an error you should inspect your Trustee KBS/AS logs and see whether there was a connection attempt for the CVM and potentially a reason why remote attestation failed.

You can set RUST_LOG=debug as environment variable the Trustee deployment to receive more verbose logs. If the evidence is being sent to KBS, the issue is most likely resolvable on the KBS side and possibly related to the evidence not meeting the expectations for a key release in KBS/AS policies or the requested resource not being available in the KBS.

If you don’t see an attestation attempt in the log of KBS there might be problems with network connectivity between the Confidential VM and the KBS. Note that the Pod and the Guest Components on the CVM might not share the same network namespace. A Pod might be able to reach KBS, but the Attestation Agent on the Confidential VM might not. If you are using HTTPS to reach the KBS, there might be a problem with the certificate provided to the Confidential VM (e.g via Initdata).

3. Retrieve decryption key from KBS

If you have successfully retrieved a token attempt to fetch the symmetric key for the encrypted image, manually and using the image’s key id (kid): kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/cdh/resource/default/images/my-key.

You can find out the kid for a given encrypted image in the a query like this:

$ ANNOTATION="org.opencontainers.image.enc.keys.provider.attestation-agent"
$ skopeo inspect docker://ghcr.io/mkulke/nginx-encrypted@sha256:5a81641ff9363a63c3f0a1417d29b527ff6e155206a720239360cc6c0722696e \
    | jq --arg ann "$ANNOTATION" -r '.LayersData[0].Annotations[$ann] | @base64d | fromjson | .kid' 
kbs:///default/image_key/nginx

If the key can be retrieved successfully, verify that the size is exactly 32 bytes and matches the key you used for encrypting the image.

4. Other (desperate) measures

If pulling an encrypted image still doesn’t work after successful retrieval of its encryption key, you might want to purge the node(s). There are bugs in the containerd remote snapshotter implementation that might taint the node with manifests and layers that will interfere with image pulling. The node should be discarded and reinstalled to rule that out.

It might also help to set annotations.io.containerd.cri.runtime-handler: $your-runtime-class in the pod spec, in addition to the runtimeClassName field and ImagePullPolicy: Always to ensure that the image is always pulled from the registry.

5 - Authenticated Registries

Use private OCI registries

Context

A user might want to use container images from private OCI registries, hence requiring authentication. The project provides the means to pull protected images from authenticated registry.

Important: ideally the authentication credentials should only be accessible from within a Trusted Execution Environment, however, due some limitations on the architecture of components used by CoCo, the credentials need to be exposed to the host, thus registry authentication is not currently a confidential feature. The community has worked to remediate that limitation and, in meanwhile, we recommend the use of encrypted images as a mitigation.

Instructions

The following steps require a functional CoCo installation on a Kubernetes cluster. A Key Broker Client (KBC) has to be configured for TEEs to be able to retrieve confidential secrets. We assume cc_kbc as a KBC for the CoCo project’s Key Broker Service (KBS) in the following instructions, but authenticated registries should work with other Key Broker implementations in a similar fashion.

Create registry authentication file

The registry authentication file should have the containers-auth.json format, with exception of credential helpers (credHelpers) that aren’t supported. Also it’s not supported glob URLs nor prefix-matched paths as in Kubernetes interpretation of config.json.

Create the registry authentication file (e.g containers-auth.json) like this:

export AUTHENTICATED_IMAGE="my-registry.local/repository/image:latest"
export AUTHENTICATED_IMAGE_NAMESPACE="$(echo "$AUTHENTICATED_IMAGE" | cut -d':' -f1)"
export AUTHENTICATED_IMAGE_USER="MyRegistryUser"
export AUTHENTICATED_IMAGE_PASSWORD="MyRegistryPassword"
cat <<EOF>> containers-auth.json
{
	"auths": {
		"${AUTHENTICATED_IMAGE_NAMESPACE}": {
			"auth": "$(echo ${AUTHENTICATED_IMAGE_USER}:${AUTHENTICATED_IMAGE_PASSWORD} | base64 -w 0)"
		}
	}
}
EOF

Where:

AUTHENTICATED_IMAGE is the full-qualified image name
AUTHENTICATED_IMAGE_NAMESPACE is the image name without the tag
AUTHENTICATED_IMAGE_USER and AUTHENTICATED_IMAGE_PASSWORD are the registry credentials user and password, respectively
auth’s value is the colon-separated user and password (user:password) credentials string encoded in base64

Provision the registry authentication file

Prior to launching a Pod the registry authentication file needs to be provisioned to the Key Broker’s repository. For a KBS deployment on Kubernetes using the local filesystem as repository storage it would work like this:

export KEY_PATH="default/containers/auth"
kubectl exec deploy/kbs -c kbs -n coco-tenant -- mkdir -p "/opt/confidential-containers/kbs/repository/$(dirname "$KEY_PATH")"
cat containers-auth.json | kubectl exec -i deploy/kbs -c kbs -n coco-tenant -- tee "/opt/confidential-containers/kbs/repository/${KEY_PATH}" > /dev/null

The CoCo infrastructure components need to cooperate with containerd and nydus-snapshotter to pull the container image from TEE. Currently the nydus-snapshotter needs to fetch the image’s metadata from registry, then authentication credentials are read from a Kubernetes secret of docker-registry type. So it should be created a secret like this:

export SECRET_NAME="cococred"
kubectl create secret docker-registry "${SECRET_NAME}" --docker-server="https://${AUTHENTICATED_IMAGE_NAMESPACE}" \
    --docker-username="${AUTHENTICATED_IMAGE_USER}" --docker-password="${AUTHENTICATED_IMAGE_PASSWORD}"

Where:

SECRET_NAME is any secret name

Launch a Pod

Create the pod yaml (e.g. pod-image-auth.yaml) like below and apply it:

export KBS_ADDRESS="172.18.0.3:31731"
export RUNTIMECLASS="kata-qemu-coco-dev"
cat <<EOF>> pod-image-auth.yaml
apiVersion: v1
kind: Pod
metadata:
  name: image-auth-feat
  annotations:
    io.containerd.cri.runtime-handler: ${RUNTIMECLASS}
    io.katacontainers.config.hypervisor.kernel_params: ' agent.image_registry_auth=kbs:///${KEY_PATH} agent.guest_components_rest_api=resource agent.aa_kbc_params=cc_kbc::http://${KBS_ADDRESS}'
spec:
  runtimeClassName: ${RUNTIMECLASS}
  containers:
    - name: test-container
      image: ${AUTHENTICATED_IMAGE}
      imagePullPolicy: Always
      command:
        - sleep
        - infinity
  imagePullSecrets:
    - name: ${SECRET_NAME}
EOF

Where:

KBS_ADDRESS is the host:port address of KBS
RUNTIMECLASS is any of available CoCo runtimeclasses (e.g. kata-qemu-tdx, kata-qemu-snp). For this example, kata-qemu-coco-dev allows to create CoCo pod on systems without confidential hardware. It should be replaced with a class matching the TEE in use.

What distinguish the pod specification for authenticated registry from a regular CoCo pod is:

the agent.image_registry_auth property in io.katacontainers.config.hypervisor.kernel_params annotation indicates the location of the registry authentication file as a resource in the KBS
the imagePullSecrets as required by nydus-snapshotter

Check the pod gets Running:

kubectl get -f pod-image-auth.yaml
NAME              READY   STATUS    RESTARTS   AGE
image-auth-feat   1/1     Running   0          2m52s

6 - Sealed Secrets

Generate and deploy protected Kubernetes secrets

Note

Sealed Secrets depend on attestation. Configure attestation before using sealed secrets.

Sealed secrets allow confidential information to be stored in the untrusted control plane. Like normal Kubernetes secrets, sealed secrets are orchestrated by the control plane and are transparently provisioned to your workload as environment variables or volumes.

Basic Usage

Here’s how you create a vault secret. There are also envelope secrets, which are described later. Vault secrets are a pointer to resource stored in a KBS, while envelope secrets are wrapped secrets that are unwrapped with a KMS.

Creating a sealed secret

There is a helper tool for sealed secrets in the Guest Components repository.

Clone the repository.

git clone https://github.com/confidential-containers/guest-components.git

Inside the guest-components directory, you can build and run the tool with Cargo.

cargo run -p confidential-data-hub --bin secret

With the tool you can create a secret.

cargo run -p confidential-data-hub --bin secret seal vault --resource-uri kbs:///your/secret/here --provider kbs

A vault secret is fulfilled by retrieving a secret from a KBS inside the guest. The locator of your secret is specified by resource-uri.

This command should return a base64 string which you will use in the next step.

Note

For vault secrets, the secret-cli tool does not upload your resource to the KBS automatically. In addition to generating the secret string, you must also upload the resource to your KBS.

Adding a sealed secret to Kubernetes

Create a secret from your secret string using kubectl.

kubectl create secret generic sealed-secret --from-literal='secret=sealed.fakejwsheader.ewogICAgInZlcnNpb24iOiAiMC4xLjAiLAogICAgInR5cGUiOiAidmF1bHQiLAogICAgIm5hbWUiOiAia2JzOi8vL2RlZmF1bHQvc2VhbGVkLXNlY3JldC90ZXN0IiwKICAgICJwcm92aWRlciI6ICJrYnMiLAogICAgInByb3ZpZGVyX3NldHRpbmdzIjoge30sCiAgICAiYW5ub3RhdGlvbnMiOiB7fQp9Cg==.fakesignature'

Note

Sealed secrets do not currently support integrity protection. This will be added in the future, but for now a fake signature and signature header are included within the secret.

When using --from-literal you provide a mapping of secret keys and values. The secret value should be the string generated in the previous step. The secret key can be whatever you want, but make sure to use the same one in future steps. This is separate from the name of the secret.

Deploying a sealed secret to a confidential workload

You can add your sealed secret to a workload yaml file.

You can expose your sealed secret as an environment variable.

apiVersion: v1
kind: Pod
metadata:
  name: sealed-secret-pod
spec:
  runtimeClassName: kata-qemu-coco-dev
  containers:
  - name: busybox
    image: quay.io/prometheus/busybox:latest
    imagePullPolicy: Always
    command: ["echo", "$PROTECTED_SECRET"]
    env:
    - name: PROTECTED_SECRET
      valueFrom:
        secretKeyRef:
          name: sealed-secret
          key: secret

You can also expose your secret as a volume.

apiVersion: v1
kind: Pod
metadata:
  name: secret-test-pod-cc
spec:
  runtimeClassName: kata
  containers:
  - name: busybox
    image: quay.io/prometheus/busybox:latest
    imagePullPolicy: Always
    command: ["cat", "/sealed/secret-value/secret"]
    volumeMounts:
        - name: sealed-secret-volume
          mountPath: "/sealed/secret-value"
  volumes:
    - name: sealed-secret-volume
      secret:
        secretName: sealed-secret

Note

Currently sealed secret volumes must be mounted in the /sealed directory.

Advanced

Envelope Secrets

You can also create envelope secrets. With envelope secrets, the secret itself is included in the secret (unlike a vault secret, which is just a pointer to a secret). In an envelope secret, the secret value is wrapped and can be unwrapped by a KMS. This allows us to support models where the key for unwrapping secrets never leaves the KMS. It also decouples the secret from the KBS.

We currently support two KMSes for envelope secrets. See specific instructions for aliyun kms and eHSM.

7 - Init-Data

Use Init-Data to inject dynamic configurations for Pods

Confidential Containers need to access configuration data that cannot be practically embedded in the OS image, such as:

URIs and certificates required to access Key Broker Services
Agent Policy that is supposed to be enforced by the policy engine
Configuration regarding image pull, such as proxies and configuration URIs.

In CoCo project this data is referred to as Init-Data. It is specified in TOML format.

Remote attestation ensures the integrity of Init-Data.

Format

A typical Init-Data TOML file looks like this:

version = "0.1.0"
algorithm = "sha256"
[data]
"policy.rego" = '''
  <contents of agent-policy>
'''
"aa.toml" = '''
  <contents of Attestation-Agent configuration TOML>
'''

"cdh.toml" = '''
  <contents of Confidential Data Hub configuration TOML>
'''

where,

version: The format version of this Init-Data TOML. Currently, ONLY version 0.1.0 is supported.
algorithm: The hash algorithm used to calculate the digest of the Init-Data TOML. This is crucial for attestation. Acceptable values are sha256, sha384 and sha512.
data: A dictionary containing concrete configurable files. Supported entries include:
- "policy.rego": Kata Agent Policy file. This helps to restrict the pod to an expected state during runtime..
- "aa.toml": Configuration file for the Attestation Agent, responsible for remote attestation.
- "cdh.toml": Configuration file for the Confidential Data Hub, responsible for in-pod confidential data flow, including image pulling, image decryption, etc.

The detailed Init-Data format definition can be found in the referenced specification.

Use-In-Attestation

In this section, we will use Init-Data to enable a Pod to perform remote attestation with a specified Trustee.

Step 1: Prepare an Init-Data TOML

Suppose we have a KBS service listening at http://1.2.3.4:8080.

The example initdata.toml is as follows. Note that we need to set the KBS URL in the following sections:

url of [token_configs.kbs] section in "aa.toml", defining the KBS for remote attestation.
url of [kbc] section in "cdh.toml", defining the KBS for accessing confidential resources. In this case, we use the same KBS for both attestation and resource provision.

version = "0.1.0"
algorithm = "sha256"
[data]
"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SetPolicyRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := true
'''
"aa.toml" = '''
[token_configs]
[token_configs.kbs]
url = "http://1.2.3.4:8080"
'''

"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "http://1.2.3.4:8080"
'''

Step 2: Embed the Init-Data in Pod YAML

We can gzip compress and base64 encode the Init-Data TOML.

initdata=$(cat initdata.toml | gzip | base64 -w0)

Then embed it in pod.yaml

cat <<EOF > pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: encrypted-image
  annotations:
    io.containerd.cri.runtime-handler: kata-qemu-tdx
    io.katacontainers.config.hypervisor.kernel_params: ' agent.guest_components_procs=confidential-data-hub'
    io.katacontainers.config.runtime.cc_init_data: ${initdata}
spec:
  runtimeClassName: kata-qemu-tdx
  containers:
    - name: test-container
      image: some-registry.io/image
      imagePullPolicy: Always
      command:
        - sleep
        - "30"
EOF

Once deployed, the pod will perform attestation with http://1.2.3.4:8080 whenever confidential resources need to be accessed.

kubectl apply -f pod.yaml

More-Use-Cases

The dynamic configuration mechanism provided by Init-Data is highly flexible. Here are some more features based on Init-Data.

8 - Image Pull Proxy

Pull containers from self-hosted registries

In today’s cloud-native environments, directly pulling container images from public registries can introduce security and performance challenges. To mitigate these issues, an image pull proxy can act as an intermediary between your environment and external image registries. This document provides a detailed method for configuring a proxy server using Init-Data for CoCo.

Configuring an Image Pull Proxy with Initdata

It is easy to configure the image pull proxy via Init-Data.

version = "0.1.0"
algorithm = "sha256"
[data]
"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SetPolicyRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := true
'''
"aa.toml" = '''
[token_configs]
[token_configs.kbs]
url = "http://1.2.3.4:8080"
'''

"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "http://1.2.3.4:8080"

[image.image_pull_proxy]

# HTTPS proxy that will be used to pull image
#
# By default this value is not set.
https_proxy = "http://127.0.0.1:5432"

# HTTP proxy that will be used to pull image
#
# By default this value is not set.
http_proxy = "http://127.0.0.1:5432"

# No proxy env that will be used to pull image.
#
# This will ensure that when we access the image registry with specified
# IPs, both `https_proxy` and `http_proxy` will not be used.
#
# If neither `https_proxy` nor `http_proxy` is not set, this field will do nothing.
#
# By default this value is not set.
no_proxy = "192.168.0.1,localhost"
'''

Note: a new section [image.image_pull_proxy] has been added to "cdh.toml", and the field https_proxy, https_proxy and no_proxy are used to configure proxy-related settings in your environment.

By configuring these settings, you can control the proxy behavior for pulling images, ensuring both security and efficiency in your container deployments. Adjust the proxy addresses and no-proxy settings as needed to fit your network configuration.

9 - Local Registries

Pull containers from self-hosted registries

In certain scenarios, there arises a need to pull container images directly from local registries, rather than relying on public repositories. This requirement may stem from the desire for reduced latency, enhanced security, or network isolation.

Local registries can be categorized into two types based on their security protocols: HTTPS-secured registries and HTTP-accessible registries.

HTTPS-Secured Registries

For registries protected by HTTPS, communication security is paramount to ensure that image transmission remains confidential and is received intact. Registry certificate configuration needs to be configured via Init-Data for secure communication.

version = "0.1.0"
algorithm = "sha256"
[data]
"policy.rego" = '''
package agent_policy

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SetPolicyRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := true
'''
"aa.toml" = '''
[token_configs]
[token_configs.kbs]
url = "http://1.2.3.4:8080"
'''

"cdh.toml" = '''
[kbc]
name = "cc_kbc"
url = "http://1.2.3.4:8080"

[image]
# To support registries with self signed certs. This config item
# is used to add extra trusted root certifications. The certificates
# must be encoded by PEM.
#
# By default this value is not set.
extra_root_certificates = ["""
-----BEGIN CERTIFICATE-----
MIIFTDCCAvugAwIBAgIBADBGBgkqhkiG9w0BAQowOaAPMA0GCWCGSAFlAwQCAgUA
oRwwGgYJKoZIhvcNAQEIMA0GCWCGSAFlAwQCAgUAogMCATCjAwIBATB7MRQwEgYD
VQQLDAtFbmdpbmVlcmluZzELMAkGA1UEBhMCVVMxFDASBgNVBAcMC1NhbnRhIENs
YXJhMQswCQYDVQQIDAJDQTEfMB0GA1UECgwWQWR2YW5jZWQgTWljcm8gRGV2aWNl
...
-----END CERTIFICATE-----
"""]
'''

Note: a new section [image] has been added to "cdh.toml", and the field extra_root_certificates includes the certificate chain of the local registry.

HTTP-Accessible Registries

TODO

10 - Protected Storage

Add protected volumes to a pod

TODO

Features

1 - Get Attestation

Warning

Enabling

Attestation

Security

2 - Get Secret Resources

Note

3 - Signed Images

Overview

Creating an Image

Creating Keys

Signing the Image

Running an Image

Setting the Security Policy for Signed Images

See Also

Cosign-GitHub Integration

4 - Encrypted Images

Context

Instructions

Note

Encrypt an image

Provision image key

Launch a Pod

Debugging

0. Launching with an unencrypted image

1. Retrieve hardware evidence from TEE

2. Perform remote attestation

3. Retrieve decryption key from KBS

4. Other (desperate) measures

5 - Authenticated Registries

Context

Instructions

Create registry authentication file

Provision the registry authentication file

Launch a Pod

6 - Sealed Secrets

Note

Basic Usage

Creating a sealed secret

Note

Adding a sealed secret to Kubernetes

Note

Deploying a sealed secret to a confidential workload

Note

Advanced

Envelope Secrets

7 - Init-Data

Format

Use-In-Attestation

Step 1: Prepare an Init-Data TOML

Step 2: Embed the Init-Data in Pod YAML

More-Use-Cases

8 - Image Pull Proxy

Configuring an Image Pull Proxy with Initdata

9 - Local Registries

HTTPS-Secured Registries

HTTP-Accessible Registries

10 - Protected Storage