In addition to running pods inside of enclaves, Confidential Containers
provides several other features that can be used to protect workloads and data.
Securing complex workloads often requires using some of these features.
Most features depend on and require attestation, which is described in the next section.
1 - Get Attestation
Workloads that request attestation evidence
Warning
This feature is disabled by default due to some subtle security considerations
which are described below.
Workloads can directly request attestation evidence.
A workload could use this evidence to carry out its own attestation protocol.
Enabling
To enable this feature, set the following parameter in the guest kernel command line.
agent.guest_components_rest_api=all
As usual, command line configurations can be added with annotations.
The API expects runtime data to be provided. The runtime data will be included in the report.
Typically this is used to bind a nonce or the hash of a key to the evidence.
Security
Enabling this feature can make your workload vulnerable to so-called “evidence factory” attacks.
By default, different CoCo workloads can have the same TCB, even if they use different container images.
This is a design feature, but it means that when the attestation report API is enabled,
two workloads could produce interchangeable attestation reports, even if they are operated by different parties.
To make sure someone else cannot generate evidence that could be used with your attestation protocol,
configure your guest to differentiate its TCB.
For example, use the init-data (which is measured) to specify the public key of the KBS.
Init-data is currently only supported with peer pods.
More information about it will be added soon.
2 - Get Secret Resources
Workloads that request resources from Trustee
Note
Requesting secret resources depends on attestation.
Configure attestation before requesting resources
While sealed secrets can be used to store encrypted secret resources
as Kubernetes Secrets and have them transparently decrypted using keys from Trustee,
a workload can also explicitly request resources via a REST API
that is automatically exposed to the pod network namespace.
For example, you can run this command from your container.
In this example Trustee will fulfill the request for kbs:///default/key/1
assuming that the resource has been provisioned.
3 - Signed Images
Procedures to generate and deploy signed OCI images with CoCo
Overview
Encrypted images provide confidentiality,
but they do not provide authenticity or integrity. Image signatures provide
this additional property, preventing certain types of image tampering,
for example.
In this brief guide, we show two tools that can be used to sign container images:
cosign and
skopeo. The skopeo tool can be
used to create both cosign signatures or “simple signatures” (which leverage
gpg keys). For our purposes, our skopeo examples will use the simple signing
approach. In any case, the general approach is to
Create keys for signing,
Sign a newly tagged image, and
Update the KBS with the public
signature key and a security policy.
Creating an Image
Creating Keys
Create a keypair using one of two approaches, cosign or Simple Signing - gpg.
To generate a public-private keypair with cosign, provide your
COSIGN_PASSWORD and use the generate-key-pair action:
Sign the image using one of two approaches, cosign or Simple Signing - skopeo.
Use the private key to sign an image.
In this example, we assume that there is a Dockerfile (your_dockerfile below)
for creating an image that you want signed. The workflow is to build the image,
push it to ghcr (which requires docker login), and sign it.
Ensure you have a gpg key owned by the user signing the image. See the previous
subsection for generating and importing gpg keys.
Sign the image. For example, the following command uses an insecure-policy
to sign a local image called confidential-containers/test-container. It uses
the unsigned tag, and in the process of signing it, creates a new
simple-signed tag.
In this example, the resulting image is pushed to ghcr, which requires docker login:
Running a workload with a signed image is very similar to running workloads
with unsigned images. The main difference is that for a signed image, you must
provide some details to the KBS. This
security policy tells the KBS which image is signed, the type of its key
signature, and where to find the public key for verifying the signature.
Beyond this, you run the workload as you usually would (e.g. via kubectl apply).
Setting the Security Policy for Signed Images
Register the public key to KBS storage. For example:
A good tutorial for cosign and github integration is
here.
The approach is automated and targets real-world usage.
For example, this key-generation step automatically
uploads the public key, private key, and key secret to the github repo:
Procedures to encrypt and consume OCI images in a TEE
Context
A user might want to bundle sensitive data on an OCI (Docker) image. The image layers should only be accessible within a Trusted Execution Environment (TEE).
The project provides the means to encrypt an image with a symmetric key that is released to the TEE only after successful verification and appraisal in a Remote Attestation process. CoCo infrastructure components within the TEE will transparently decrypt the image layers as they are pulled from a registry without exposing the decrypted data outside the boundaries of the TEE.
Instructions
The following steps require a functional CoCo installation on a Kubernetes cluster. A Key Broker Client (KBC) has to be configured for TEEs to be able to retrieve confidential secrets. We assume cc_kbc as a KBC for the CoCo project’s Key Broker Service (KBS) in the following instructions, but image encryption should work with other Key Broker implementations in a similar fashion.
Note
Please ensure you have a recent version of Skopeo (v1.13.3+) installed locally.
Encrypt an image
We extend public image with secret data.
docker build -t unencrypted - <<EOF
FROM nginx:stable
RUN echo "something confidential" > /secret
EOF
The encryption key needs to be a 32 byte sequence and provided to the encryption step as base64-encoded string.
The key id is a generic resource descriptor used by the key broker to look up secrets in its storage. For KBS this is composed of three segments: $repository_name/$resource_type/$resource_tag
To access the image from within the container, Skopeo can be used to buffer the image in a directory, which is then made available to the container. Similarly, the resulting encrypted image will be put into an output directory.
Prior to launching a Pod the image key needs to be provisioned to the Key Broker’s repository. For a KBS deployment on Kubernetes using the local filesystem as repository storage it would work like this:
Note: If you’re not using KBS deployment using trustee operator additional namespace may be needed -n coco-tenant.
Launch a Pod
We create a simple deployment using our encrypted image. As the image is being pulled and the CoCo components in the TEE encounter the layer annotations that we saw above, the image key will be retrieved from the Key Broker using the annotated Key ID and the layers will be decrypted transparently and the container should come up.
In this example we default to the Cloud API Adaptor runtime, adjust this depending on the CoCo installation.
kubectl get runtimeclass -o jsonpath='{.items[].handler}'
Note: If you’re not using KBS deployment using trustee operator additional namespace may be needed -n coco-tenant.
Debugging
The encrypted image feature relies on a cascade of other features as building blocks:
Image decryption is built on secret retrieval from KBS
Secret retrieval is built on remote attestation
Remote attestation requires retrieval of hardware evidence from the TEE
All of the above have to work in order to decrypt an encrypted image.
0. Launching with an unencrypted image
Launch the same image with unencrypted layers from the same registry to verify that the image itself is not an issue.
1. Retrieve hardware evidence from TEE
Launch an unencrypted library/nginx deployment named nginx with a CoCo runtime class. Issue kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/aa/evidence\?runtime_data\=xxxx. This should produce a real hardware evidence rendered as JSON. Something like {"svn":"1","report_data":"eHh4eA=="} is not real hardware evidence, it’s a dummy response from the sample attester. Real TEE evidence should be more verbose and contain certificates and measurements.
The reason for not producing real evidence could be a wrong build of Attestation Agent for the TEE that you are attempting to use, or the Confidential VM not exposing the expected interfaces.
Note: In some configurations of the CoCo image the facility to retrieve evidence is disabled by default. For bare-metal CoCo images you can enable it by setting agent.guest_components_rest_api=all on the kernel cmdline (see here).
2. Perform remote attestation
Run kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/aa/token\?token_type\=kbs. This should produce an attestation token. If you don’t receive a token but an error you should inspect your Trustee KBS/AS logs and see whether there was a connection attempt for the CVM and potentially a reason why remote attestation failed.
You can set RUST_LOG=debug as environment variable the Trustee deployment to receive more verbose logs. If the evidence is being sent to KBS, the issue is most likely resolvable on the KBS side and possibly related to the evidence not meeting the expectations for a key release in KBS/AS policies or the requested resource not being available in the KBS.
If you don’t see an attestation attempt in the log of KBS there might be problems with network connectivity between the Confidential VM and the KBS. Note that the Pod and the Guest Components on the CVM might not share the same network namespace. A Pod might be able to reach KBS, but the Attestation Agent on the Confidential VM might not. If you are using HTTPS to reach the KBS, there might be a problem with the certificate provided to the Confidential VM (e.g via Initdata).
3. Retrieve decryption key from KBS
If you have successfully retrieved a token attempt to fetch the symmetric key for the encrypted image, manually and using the image’s key id (kid): kubectl exec deploy/nginx -- curl http://127.0.0.1:8006/cdh/resource/default/images/my-key.
You can find out the kid for a given encrypted image in the a query like this:
If the key can be retrieved successfully, verify that the size is exactly 32 bytes and matches the key you used for encrypting the image.
4. Other (desperate) measures
If pulling an encrypted image still doesn’t work after successful retrieval of its encryption key, you might want to purge the node(s). There are bugs in the containerd remote snapshotter implementation that might taint the node with manifests and layers that will interfere with image pulling. The node should be discarded and reinstalled to rule that out.
It might also help to set annotations.io.containerd.cri.runtime-handler: $your-runtime-class in the pod spec, in addition to the runtimeClassName field and ImagePullPolicy: Always to ensure that the image is always pulled from the registry.
5 - Authenticated Registries
Use private OCI registries
Context
A user might want to use container images from private OCI registries, hence requiring authentication. The project provides the means to pull protected
images from authenticated registry.
Important: ideally the authentication credentials should only be accessible from within a Trusted Execution Environment, however, due some limitations
on the architecture of components used by CoCo, the credentials need to be exposed to the host, thus registry authentication is not currently a confidential feature. The community has worked to remediate that limitation and, in meanwhile, we recommend the use of encrypted images as a mitigation.
Instructions
The following steps require a functional CoCo installation on a Kubernetes cluster. A Key Broker Client (KBC) has to be configured for TEEs to be able to retrieve confidential secrets. We assume cc_kbc as a KBC for the CoCo project’s Key Broker Service (KBS) in the following instructions, but authenticated registries should work with other Key Broker implementations in a similar fashion.
Create registry authentication file
The registry authentication file should have the containers-auth.json format, with exception of credential helpers (credHelpers) that aren’t supported. Also it’s not supported glob URLs nor prefix-matched paths as in
Kubernetes interpretation of config.json.
Create the registry authentication file (e.g containers-auth.json) like this:
AUTHENTICATED_IMAGE is the full-qualified image name
AUTHENTICATED_IMAGE_NAMESPACE is the image name without the tag
AUTHENTICATED_IMAGE_USER and AUTHENTICATED_IMAGE_PASSWORD are the registry credentials user and password, respectively
auth’s value is the colon-separated user and password (user:password) credentials string encoded in base64
Provision the registry authentication file
Prior to launching a Pod the registry authentication file needs to be provisioned to the Key Broker’s repository. For a KBS deployment on Kubernetes using the local filesystem as repository storage it would work like this:
exportKEY_PATH="default/containers/auth"kubectl exec deploy/kbs -c kbs -n coco-tenant -- mkdir -p "/opt/confidential-containers/kbs/repository/$(dirname "$KEY_PATH")"cat containers-auth.json | kubectl exec -i deploy/kbs -c kbs -n coco-tenant -- tee "/opt/confidential-containers/kbs/repository/${KEY_PATH}" > /dev/null
The CoCo infrastructure components need to cooperate with containerd and nydus-snapshotter to pull the container image from TEE. Currently
the nydus-snapshotter needs to fetch the image’s metadata from registry, then authentication credentials are read from a Kubernetes secret
of docker-registry type. So it should be created a secret like this:
RUNTIMECLASS is any of available CoCo runtimeclasses (e.g. kata-qemu-tdx, kata-qemu-snp). For this example, kata-qemu-coco-dev allows to create CoCo pod on systems without confidential hardware. It should be replaced with a class matching the TEE in use.
What distinguish the pod specification for authenticated registry from a regular CoCo pod is:
the agent.image_registry_auth property in io.katacontainers.config.hypervisor.kernel_params annotation indicates the location of the registry authentication file as a resource in the KBS
the imagePullSecrets as required by nydus-snapshotter
Check the pod gets Running:
kubectl get -f pod-image-auth.yaml
NAME READY STATUS RESTARTS AGE
image-auth-feat 1/1 Running 0 2m52s
6 - Sealed Secrets
Generate and deploy protected Kubernetes secrets
Note
Sealed Secrets depend on attestation.
Configure attestation before using sealed secrets.
Sealed secrets allow confidential information to be stored in the untrusted control plane.
Like normal Kubernetes secrets, sealed secrets are orchestrated by the control plane
and are transparently provisioned to your workload as environment variables or volumes.
Basic Usage
Here’s how you create a vault secret.
There are also envelope secrets, which are described later.
Vault secrets are a pointer to resource stored in a KBS,
while envelope secrets are wrapped secrets that are unwrapped with a KMS.
Creating a sealed secret
There is a helper tool for sealed secrets in the Guest Components repository.
Inside the guest-components directory, you can build and run the tool with Cargo.
cargo run -p confidential-data-hub --bin secret
With the tool you can create a secret.
cargo run -p confidential-data-hub --bin secret seal vault --resource-uri kbs:///your/secret/here --provider kbs
A vault secret is fulfilled by retrieving a secret from a KBS inside the guest.
The locator of your secret is specified by resource-uri.
This command should return a base64 string which you will use in the next step.
Note
For vault secrets, the secret-cli tool does not upload your resource to the KBS
automatically.
In addition to generating the secret string, you must also upload the resource
to your KBS.
Adding a sealed secret to Kubernetes
Create a secret from your secret string using kubectl.
Sealed secrets do not currently support integrity protection.
This will be added in the future, but for now a fake signature
and signature header are included within the secret.
When using --from-literal you provide a mapping of secret keys and values.
The secret value should be the string generated in the previous step.
The secret key can be whatever you want, but make sure to use the same one in future steps.
This is separate from the name of the secret.
Deploying a sealed secret to a confidential workload
You can add your sealed secret to a workload yaml file.
You can expose your sealed secret as an environment variable.
Currently sealed secret volumes must be mounted
in the /sealed directory.
Advanced
Envelope Secrets
You can also create envelope secrets.
With envelope secrets, the secret itself is included in the secret
(unlike a vault secret, which is just a pointer to a secret).
In an envelope secret, the secret value is wrapped and can be unwrapped
by a KMS.
This allows us to support models where the key for unwrapping secrets
never leaves the KMS.
It also decouples the secret from the KBS.
We currently support two KMSes for envelope secrets.
See specific instructions for aliyun kms
and eHSM.
7 - Init-Data
Use Init-Data to inject dynamic configurations for Pods
Confidential Containers need to access configuration data that cannot be practically embedded in the OS image, such as:
URIs and certificates required to access Key Broker Services
Agent Policy that is supposed to be enforced by the policy engine
Configuration regarding image pull, such as proxies and configuration URIs.
In CoCo project this data is referred to as Init-Data. It is specified in TOML format.
Remote attestation ensures the integrity of Init-Data.
Format
A typical Init-Data TOML file looks like this:
version="0.1.0"algorithm="sha256"[data]"policy.rego"='''
<contents of agent-policy>
'''"aa.toml"='''
<contents of Attestation-Agent configuration TOML>
'''"cdh.toml"='''
<contents of Confidential Data Hub configuration TOML>
'''
where,
version: The format version of this Init-Data TOML. Currently, ONLY version 0.1.0 is supported.
algorithm: The hash algorithm used to calculate the digest of the Init-Data TOML. This is crucial for attestation. Acceptable values are sha256, sha384 and sha512.
data: A dictionary containing concrete configurable files. Supported entries include:
"policy.rego": Kata Agent Policy file. This helps to restrict the pod to an expected state during runtime..
"cdh.toml": Configuration file for the Confidential Data Hub, responsible for in-pod confidential data flow, including image pulling, image decryption, etc.
In this section, we will use Init-Data to enable a Pod to perform remote attestation with a specified Trustee.
Step 1: Prepare an Init-Data TOML
Suppose we have a KBS service listening at http://1.2.3.4:8080.
The example initdata.toml is as follows. Note that we need to set the KBS URL in the following sections:
url of [token_configs.kbs] section in "aa.toml", defining the KBS for remote attestation.
url of [kbc] section in "cdh.toml", defining the KBS for accessing confidential resources. In this case, we use the same KBS for both attestation and resource provision.
In today’s cloud-native environments, directly pulling container images from public registries can introduce security and performance challenges. To mitigate these issues, an image pull proxy can act as an intermediary between your environment and external image registries. This document provides a detailed method for configuring a proxy server using Init-Data for CoCo.
Configuring an Image Pull Proxy with Initdata
It is easy to configure the image pull proxy via Init-Data.
version="0.1.0"algorithm="sha256"[data]"policy.rego"='''
package agent_policy
default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default ExecProcessRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default ReadStreamRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SetPolicyRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := true
'''"aa.toml"='''
[token_configs]
[token_configs.kbs]
url = "http://1.2.3.4:8080"
'''"cdh.toml"='''
[kbc]
name = "cc_kbc"
url = "http://1.2.3.4:8080"
[image.image_pull_proxy]
# HTTPS proxy that will be used to pull image
#
# By default this value is not set.
https_proxy = "http://127.0.0.1:5432"
# HTTP proxy that will be used to pull image
#
# By default this value is not set.
http_proxy = "http://127.0.0.1:5432"
# No proxy env that will be used to pull image.
#
# This will ensure that when we access the image registry with specified
# IPs, both `https_proxy` and `http_proxy` will not be used.
#
# If neither `https_proxy` nor `http_proxy` is not set, this field will do nothing.
#
# By default this value is not set.
no_proxy = "192.168.0.1,localhost"
'''
Note: a new section [image.image_pull_proxy] has been added to "cdh.toml", and the field https_proxy, https_proxy and no_proxy are used to configure proxy-related settings in your environment.
By configuring these settings, you can control the proxy behavior for pulling images, ensuring both security and efficiency in your container deployments. Adjust the proxy addresses and no-proxy settings as needed to fit your network configuration.
9 - Local Registries
Pull containers from self-hosted registries
In certain scenarios, there arises a need to pull container images directly from local registries, rather than relying on public repositories. This requirement may stem from the desire for reduced latency, enhanced security, or network isolation.
Local registries can be categorized into two types based on their security protocols: HTTPS-secured registries and HTTP-accessible registries.
HTTPS-Secured Registries
For registries protected by HTTPS, communication security is paramount to ensure that image transmission remains confidential and is received intact. Registry certificate configuration needs to be configured via Init-Data for secure communication.