Blog

• Releases
• Release v0.8.0
• Release v0.7.0
• Release v0.6.0
• Release v0.5.0
• Release v0.4.0
• Release v0.3.0
• Release v0.2.0
• Release v0.1.0
• Memory Protection for AI ML Model Inferencing
• Building Trust into OS images for Confidential Containers
• Introduction to Confidential Containers (CoCo)

Releases

Release v0.8.0

Please see the quickstart guide for details on how to try out Confidential Containers.

Please refer to our Acronyms and Glossary pages for a definition of the acronyms used in this document.

What’s new

• Upstream containerd supported by all deployment types except enclave-cc.
  • This release includes the Nydus snapshotter (for the first time) to support upstream containerd.
  • In this release images are still pulled inside the guest.
  • Nydus snapshotter requires the following annotation for each pod io.containerd.cri.runtime-handler: <runtime-class>.
  • Support for Nydus snapshotter in peer pods is still experimental. To avoid using it with peer pods do not set above annotation.
  • Nydus snapshotter support in general is still evolving. See limitations section below for details.
• A new component, the Confidential Data Hub (CDH) is now deployed inside the guest.
  • CDH is an evolution of the Attestation Agent that supports advanced features.
  • CDH supports sealed Kubernetes secrets which are managed by the control plane, but securely unwrapped inside the enclave.
  • CDH supports connections to both KBS and KMS.
• New architecture of Attestation Agent and CDH allows a client to deploy multiple KBSes.
  • One KBS can be used for validating evidence with the Attestation Service while another can provide resources.
• Pulling from an authenticated registry now requires imagePullSecrets.

Peer Pods

• peerpod-ctl tool has been expanded.
  • Can check and clean old peerpod objects
  • Adds SSH authentication support to libvirt provider
  • Supports IBM cloud
• Support for secure key release at runtime and image decryption via remote attestation on AKS
• Added AMD SEV and IBM s390x support for the Libvirt provider
• Container registry authentication now bootstrapped from userdata.
• Enabled public IP usage for pod VM on AWS and PowerVS providers
• webhook: added IBM ppc64le platform support
• Support adding custom tags to podvm instances
• Switched to launching CVM by default on AWS and Azure providers
• Added rollingUpdate strategy in cloud-api-adaptor daemonset
• Disabled secureboot by default

Hardware Support

Confidential Containers is tested with attestation on the following platforms:

• Intel TDX
• AMD SEV(-ES)
• Intel SGX

The following platforms are untested or partially supported:

• IBM Secure Execution (SE) on IBM zSystems (s390x) running LinuxONE
• AMD SEV-SNP
• ARM CCA

Limitations

The following are known limitations of this release:

• Nydus snapshotter support is not mature.
  • Nydus snapshot sometimes conflicts with existing node configuration.
  • You may need to remove existing container images/snapshots before installing Nydus snapshotter.
  • Nydus snapshotter may not support pulling one image with multiple runtime handler annotations even across different pods.
  • Host pulling with Nydus snapshotter is not yet enabled.
  • Nydus snapshotter is not supported with enclave-cc.
• Pulling container images inside guest may have negative performance implications including greater resource usage and slower startup.
• crio support is still evolving.
• Platform support is rapidly changing
  • Image signature validation with AMD SEV-ES is not covered by CI.
• SELinux is not supported on the host and must be set to permissive if in use.
• The generic KBS does not yet supported all platforms.
• The format of encrypted container images is still subject to change
  • The oci-crypt container image format itself may still change
  • The tools to generate images are not in their final form
  • The image format itself is subject to change in upcoming releases
  • Not all image repositories support encrypted container images. Complete integration with Kubernetes is still in progress.
  • OpenShift support is not yet complete.
  • Existing APIs do not fully support the CoCo security and threat model. More info
  • Some commands accessing confidential data, such as kubectl exec, may either fail to work, or incorrectly expose information to the host
• The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
  • We track our status with the OpenSSF Best Practices Badge, which improved to 69% at the time of this release.
  • Vulnerability reporting mechanisms still need to be created. Public github issues are still appropriate for this release until private reporting is established.
• Container metadata such as environment variables are not measured.
• Kata Agent does not validate mount requests. A malicious host might be able to mount a shared filesystem into the PodVM.

CVE Fixes

None

Release v0.7.0

Please see the quickstart guide for details on how to try out Confidential Containers.

Please refer to our Acronyms and Glossary pages for a definition of the acronyms used in this document.

What’s new

• Flexible instance types/profiles support for peer-pods
• Ability to use CSI Persistent Volume with peer-pods on Azure and IBM Cloud
• EAA-KBC/Verdictd support removed from enclave-cc
• Baremetal SNP without attestation available via operator
• Guest components (attestation-agent, image-rs and ocicrypt-rs) merged into one repository
• Documentation and community repositories merged together

Hardware Support

Confidential Containers is tested with attestation on the following platforms:

• Intel TDX
• AMD SEV(-ES)
• Intel SGX

The following platforms are untested or partially supported:

• IBM Secure Execution (SE) on IBM zSystems (s390x) running LinuxONE
• AMD SEV-SNP

The following platforms are in development:

• ARM CCA

Limitations

The following are known limitations of this release:

• Platform support is rapidly changing
  • Image signature validation with AMD SEV-ES is not covered by CI.
• SELinux is not supported on the host and must be set to permissive if in use.
• The generic KBS does not yet supported all platforms.
• The format of encrypted container images is still subject to change
  • The oci-crypt container image format itself may still change
  • The tools to generate images are not in their final form
  • The image format itself is subject to change in upcoming releases
  • Not all image repositories support encrypted container images.
• CoCo currently requires a custom build of containerd, which is installed by the operator.
  • Codepath for pulling images will change significantly in future releases.
  • crio is only supported with cloud-api-adaptor.
• Complete integration with Kubernetes is still in progress.
  • OpenShift support is not yet complete.
  • Existing APIs do not fully support the CoCo security and threat model. More info
  • Some commands accessing confidential data, such as kubectl exec, may either fail to work, or incorrectly expose information to the host
  • Container images must be downloaded separately (inside guest) for each pod. More info
• The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
  • We track our status with the OpenSSF Best Practices Badge, which remained at 64% at the time of this release.
  • Vulnerability reporting mechanisms still need to be created. Public github issues are still appropriate for this release until private reporting is established.

CVE Fixes

None

Release v0.6.0

Please see the quickstart guide for details on how to try out Confidential Containers.

Please refer to our Acronyms and Glossary pages for a definition of the acronyms used in this document.

What’s new

• Support for attesting pod VMs with Azure vTPMs on SEV-SNP
• Support for using Project Amber as an attestation service
• Support for Cosign signature validation with s390x
• Pulling guest images with many layers can no longer cause guest CPU starvation.
• Attestation Service upgraded to avoid several security issues in Go packages.
• CC-KBC & KBS support with SGX attester/verifier for Occlum and CI for enclave-cc

Hardware Support

Confidential Containers is tested with attestation on the following platforms:

• Intel TDX
• AMD SEV(-ES)
• Intel SGX

The following platforms are untested or partially supported:

• IBM Secure Execution (SE) on IBM zSystems (s390x) running LinuxONE
• AMD SEV-SNP

The following platforms are in development:

• ARM CCA

Limitations

The following are known limitations of this release:

• Platform support is rapidly changing
  • Image signature validation with AMD SEV-ES is not covered by CI.
• SELinux is not supported on the host and must be set to permissive if in use.
• The generic KBS does not yet supported all platforms.
• The format of encrypted container images is still subject to change
  • The oci-crypt container image format itself may still change
  • The tools to generate images are not in their final form
  • The image format itself is subject to change in upcoming releases
  • Not all image repositories support encrypted container images.
• CoCo currently requires a custom build of containerd, which is installed by the operator.
  • Codepath for pulling images will change significantly in future releases.
  • crio is only supported with cloud-api-adaptor.
• Complete integration with Kubernetes is still in progress.
  • OpenShift support is not yet complete.
  • Existing APIs do not fully support the CoCo security and threat model. More info
  • Some commands accessing confidential data, such as kubectl exec, may either fail to work, or incorrectly expose information to the host
  • Container images must be downloaded separately (inside guest) for each pod. More info
• The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
  • We track our status with the OpenSSF Best Practices Badge, which remained at 64% at the time of this release.
  • Vulnerability reporting mechanisms still need to be created. Public github issues are still appropriate for this release until private reporting is established.

CVE Fixes

None

Release v0.5.0

Please see the quickstart guide for details on how to try out Confidential Containers.

Please refer to our Acronyms and Glossary pages for a definition of the acronyms used in this document.

What’s new

• Process-based isolation is now fully supported with SGX hardware added to enclave-cc CI

• Remote hypervisor support added to the CoCo operator, which helps to enable creating containers as ‘peer pods’, either locally, or on Cloud Service Provider Infrastructure. See README for more information and installation instructions.

• KBS Resource URI Scheme is published to identify all confidential resources.

• Different KBCs now share image encryption format allowing for interchangeable use.

• Generic Key Broker System (KBS) is now supported. This includes the KBS itself, which relies on the Attestation Service (AS) for attestation evidence verification. Reference Values are provided to the AS by the Reference Value Provider Service (RVPS). Currently only TDX and a sample mode are supported with generic KBS. Other platforms are in development.

• SEV configuration can be set with annotations.

• SEV-ES is now tested in the CI.

• Some developmental SEV-SNP components can be manually enabled to test SNP containers without attestation.

Hardware Support

Confidential Containers is tested with attestation on the following platforms:

• Intel TDX
• AMD SEV(-ES)
• Intel SGX

The following platforms are untested or partially supported:

• IBM Secure Execution (SE) on IBM zSystems (s390x) running LinuxONE

The following platforms are in development:

• AMD SEV-SNP

Limitations

The following are known limitations of this release:

• Platform support is currently limited, and rapidly changing
  • Image signature validation with AMD SEV-ES is not covered by CI.
  • s390x does not support cosign signature validation
• SELinux is not supported on the host and must be set to permissive if in use.
• Attestation and key brokering support varies by platform.
  • The generic KBS is only supported on TDX. Other platforms have different solutions.
• The format of encrypted container images is still subject to change
  • The oci-crypt container image format itself may still change
  • The tools to generate images are not in their final form
  • The image format itself is subject to change in upcoming releases
  • Image repository support for encrypted images is unequal
• CoCo currently requires a custom build of containerd
  • The CoCo operator will deploy the correct version of containerd for you
  • Changes are required to delegate PullImage to the agent in the virtual machine
  • The required changes are not part of the vanilla containerd
  • The final form of the required changes in containerd is expected to be different
  • crio is not supported
• CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift)
  • OpenShift support is not yet complete.
  • Existing APIs do not fully support the CoCo security and threat model. More info
  • Some commands accessing confidential data, such as kubectl exec, may either fail to work, or incorrectly expose information to the host
  • Container image sharing is not possible in this release
  • Container images are downloaded by the guest (with encryption), not by the host
  • As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. More info
• The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
  • We track our status with the OpenSSF Best Practices Badge, which increased from 49% to 64% at the time of this release.
  • All CoCo repos now have automated tests, including linting, incorporated into CI.
  • Vulnerability reporting mechanisms still need to be created. Public github issues are still appropriate for this release until private reporting is established.

CVE Fixes

None

Release v0.4.0

Please see the quickstart guide for details on how to try out Confidential Containers.

Please refer to our Acronyms and Glossary pages for a definition of the acronyms used in this document.

What’s new

• This release focused on reducing technical debt. You will not observe as many new features in this release but you will be running on top of more robust code.
• Skopeo and umoci dependencies are removed with our image-rs component fully integrated
• Improved CI for SEV
• Improved container support for enclave-cc / SGX

Hardware Support

Confidential Containers is tested with attestation on the following platforms:

• Intel TDX
• AMD SEV

The following platforms are untested or partially supported:

• Intel SGX
• AMD SEV-ES
• IBM Secure Execution (SE) on IBM zSystems (s390x) running LinuxONE

The following platforms are in development:

• AMD SEV-SNP

Limitations

The following are known limitations of this release:

• Platform support is currently limited, and rapidly changing
  • AMD SEV-ES is not tested in the CI.
  • Image signature validation has not been tested with AMD SEV.
  • s390x does not support cosign signature validation
• SELinux is not supported on the host and must be set to permissive if in use.
• Attestation and key brokering support is still under development
  • The disk-based key broker client (KBC) is used for non-tee testing, but is not suitable for production, except with encrypted VM images.
  • Currently, there are two key broker services (KBS) that can be used:
    • simple-kbs: simple key broker service for SEV(-ES).
    • Verdictd: An external project with which Attestation Agent can conduct remote attestation communication and key acquisition via EAA KBC
  • The full-featured generic KBS and the corresponding KBC are still in the development stage.
• The format of encrypted container images is still subject to change
  • The oci-crypt container image format itself may still change
  • The tools to generate images are not in their final form
  • The image format itself is subject to change in upcoming releases
  • Image repository support for encrypted images is unequal
• CoCo currently requires a custom build of containerd
  • The CoCo operator will deploy the correct version of containerd for you
  • Changes are required to delegate PullImage to the agent in the virtual machine
  • The required changes are not part of the vanilla containerd
  • The final form of the required changes in containerd is expected to be different
  • crio is not supported
• CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift)
  • OpenShift is a non-starter at the moment due to its dependency on CRI-O
  • Existing APIs do not fully support the CoCo security and threat model. More info
  • Some commands accessing confidential data, such as kubectl exec, may either fail to work, or incorrectly expose information to the host
  • Container image sharing is not possible in this release
  • Container images are downloaded by the guest (with encryption), not by the host
  • As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. More info
• The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
  • We track our status with the OpenSSF Best Practices Badge, which increased to 49% at the time of this release.
  • The main gaps are in test coverage, both general and security tests.
  • Vulnerability reporting mechanisms also need to be created. Public github issues are still appropriate for this release until private reporting is established.

CVE Fixes

None

Release v0.3.0

Code Freeze: January 13th, 2023

Please see the quickstart guide for details on how to try out Confidential Containers

What’s new

• Support for pulling images from authenticated container registries. See design info.
• Significantly reduced resource requirements for image pulling
• Attestation support for AMD SEV-ES
• kata-qemu-tdx supports and has been tested with Verdictd
• Support for get_resource endpoint with SEV(-ES)
• Enabled cosign signature support in enclave-cc / SGX
• SEV attestation bug fixes
• Measured rootfs now works with kata-clh, kata-qemu, kata-clh-tdx, and kata-qemu-tdx runtime classes.
• IBM zSystems / LinuxONE (s390x) enablement and CI verification on non-TEE environments
• Enhanced docs, config, CI pipeline and test coverage for enclave-cc / SGX

Hardware Support

Confidential Containers is tested with attestation on the following platforms:

• Intel TDX
• AMD SEV

The following platforms are untested or partially supported:

• Intel SGX
• AMD SEV-ES
• IBM Secure Execution (SE) on IBM zSystems & LinuxONE

The following platforms are in development:

• AMD SEV-SNP

Limitations

The following are known limitations of this release:

• Platform support is currently limited, and rapidly changing
  • AMD SEV-ES is not tested in the CI.
  • Image signature validation has not been tested with AMD SEV.
  • s390x does not support cosign signature validation
• SELinux is not supported on the host and must be set to permissive if in use.
• Attestation and key brokering support is still under development
  • The disk-based key broker client (KBC) is used for non-tee testing, but is not suitable for production, except with encrypted VM images.
  • Currently, there are two KBS that can be used:
    • simple-kbs: simple key broker service (KBS) for SEV(-ES).
    • Verdictd: An external project with which Attestation Agent can conduct remote attestation communication and key acquisition via EAA KBC
  • The full-featured generic KBS and the corresponding KBC are still in the development stage.
  • For developers, other KBCs can be experimented with.
  • AMD SEV must use a KBS even for unencrypted images.
• The format of encrypted container images is still subject to change
  • The oci-crypt container image format itself may still change
  • The tools to generate images are not in their final form
  • The image format itself is subject to change in upcoming releases
  • Image repository support for encrypted images is unequal
• CoCo currently requires a custom build of containerd
  • The CoCo operator will deploy the correct version of containerd for you
  • Changes are required to delegate PullImage to the agent in the virtual machine
  • The required changes are not part of the vanilla containerd
  • The final form of the required changes in containerd is expected to be different
  • crio is not supported
• CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift)
  • OpenShift is a non-starter at the moment due to its dependency on CRI-O
  • Existing APIs do not fully support the CoCo security and threat model. More info
  • Some commands accessing confidential data, such as kubectl exec, may either fail to work, or incorrectly expose information to the host
  • Container image sharing is not possible in this release
  • Container images are downloaded by the guest (with encryption), not by the host
  • As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. More info
• The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
  • We track our status with the OpenSSF Best Practices Badge, which increased to 49% at the time of this release.
  • The main gaps are in test coverage, both general and security tests.
  • Vulnerability reporting mechanisms also need to be created. Public github issues are still appropriate for this release until private reporting is established.

CVE Fixes

None

Release v0.2.0

Confidential Containers has adopted a six-week release cadence. This is our first release on this schedule. This release mainly features incremental improvements to our build system and tests as well as minor features, adjustments, and cleanup.

Please see the quickstart guide for details on how to try out Confidential Containers

What’s new

• Kata CI uses existing Kata tooling to build components.
• Kata CI caches build environments for components.
• Pod VM can be launched with measured boot. See more info
• Incremental advances in signature support including verification of cosign-signed images.
• Enclave-cc added to operator, providing initial SGX support.
• KBS no longer required to use unencrypted images with SEV.
• More rigorous versioning of sub-projects

Hardware Support

Confidential Containers is tested with attestation on the following platforms:

• Intel TDX
• AMD SEV

The following platforms are untested or partially supported:

• Intel SGX
• AMD SEV-ES
• IBM Z SE

The following platforms are in development:

• AMD SEV-SNP

Limitations

The following are known limitations of this release:

• Platform support is currently limited, and rapidly changing
  • s390x is not supported by the CoCo operator
  • AMD SEV-ES has not been tested.
  • AMD SEV does not support container image signature validation.
  • s390x does not support cosign signature validation
• SELinux is not supported on the host and must be set to permissive if in use.
• Attestation and key brokering support is still under development
  • The disk-based key broker client (KBC) is used for non-tee testing, but is not suitable for production, except with encrypted VM images.
  • Currently, there are two KBS that can be used:
    • simple-kbs: simple key broker service (KBS) for SEV(-ES).
    • Verdictd: An external project with which Attestation Agent can conduct remote attestation communication and key acquisition via EAA KBC
  • The full-featured generic KBS and the corresponding KBC are still in the development stage.
  • For developers, other KBCs can be experimented with.
  • AMD SEV must use a KBS even for unencrypted images.
• The format of encrypted container images is still subject to change
  • The oci-crypt container image format itself may still change
  • The tools to generate images are not in their final form
  • The image format itself is subject to change in upcoming releases
  • Image repository support for encrypted images is unequal
• CoCo currently requires a custom build of containerd
  • The CoCo operator will deploy the correct version of containerd for you
  • Changes are required to delegate PullImage to the agent in the virtual machine
  • The required changes are not part of the vanilla containerd
  • The final form of the required changes in containerd is expected to be different
  • crio is not supported
• CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift)
  • OpenShift is a non-starter at the moment due to its dependency on CRI-O
  • Existing APIs do not fully support the CoCo security and threat model. More info
  • Some commands accessing confidential data, such as kubectl exec, may either fail to work, or incorrectly expose information to the host
  • Container image sharing is not possible in this release
  • Container images are downloaded by the guest (with encryption), not by the host
  • As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host. More info
• The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
  • We track our status with the OpenSSF Best Practices Badge, which increased to 46% at the time of this release.
  • The main gaps are in test coverage, both general and security tests.
  • Vulnerability reporting mechanisms also need to be created. Public github issues are still appropriate for this release until private reporting is established.

CVE Fixes

None

Release v0.1.0

This is the first full release of Confidential Containers. The goal of this release is to provide a stable, simple, and well-documented base for the Confidential Containers project. The Confidential Containers operator is the focal point of the release. The operator allows users to install Confidential Containers on an existing Kubernetes cluster. This release also provides core Confidential Containers features, such as being able to run encrypted containers on Intel-TDX and AMD-SEV.

Please see the quickstart guide for details on how to try out Confidential Containers"

Hardware Support

Confidential Containers is tested with attestation on the following platforms:

• Intel TDX
• AMD SEV

The following platforms are untested or partially supported:

• AMD SEV-ES
• IBM Z SE

The following platforms are in development:

• Intel SGX
• AMD SEV-SNP

Limitations

The following are known limitations of this release:

• Platform support is currently limited, and rapidly changing
  • S390x is not supported by the CoCo operator
  • AMD SEV-ES has not been tested.
  • AMD SEV does not support container image signature validation.
• Attestation and key brokering support is still under development
  • The disk-based key broker client (KBC) is used when there is no HW support, but is not suitable for production (except with encrypted VM images).
  • Currently, there are two KBS that can be used:
    • simple-kbs: simple key broker service (KBS) for SEV(-ES).
    • Verdictd: An external project with which Attestation Agent can conduct remote attestation communication and key acquisition via EAA KBC
  • The full-featured generic KBS and the corresponding KBC are still in the development stage.
  • For developers, other KBCs can be experimented with.
  • AMD SEV must use a KBS even for unencrypted images.
• The format of encrypted container images is still subject to change
  • The oci-crypt container image format itself may still change
  • The tools to generate images are not in their final form
  • The image format itself is subject to change in upcoming releases
  • Image repository support for encrypted images is unequal
• CoCo currently requires a custom build of containerd
  • The CoCo operator will deploy the correct version of containerd for you
  • Changes are required to delegate PullImage to the agent in the virtual machine
  • The required changes are not part of the vanilla containerd
  • The final form of the required changes in containerd is expected to be different
  • crio is not supported
• CoCo is not fully integrated with the orchestration ecosystem (Kubernetes, OpenShift)
  • OpenShift is a non-started at the moment due to their dependency on CRIO
  • Existing APIs do not fully support the CoCo security and threat model
  • Some commands accessing confidential data, such as kubectl exec, may either fail to work, or incorrectly expose information to the host
  • Container image sharing is not possible in this release
  • Container images are downloaded by the guest (with encryption), not by the host
  • As a result, the same image will be downloaded separately by every pod using it, not shared between pods on the same host.
• The CoCo community aspires to adopting open source security best practices, but not all practices are adopted yet.
  • We track our status with the OpenSSF Best Practices Badge, which was at 43% at the time of this release.
  • The main gaps are in test coverage, both general and security tests.
  • Vulnerability reporting mechanisms also need to be created. Public github issues are still appropriate for this release until private reporting is established.

CVE Fixes

None - This is our first release.

Memory Protection for AI ML Model Inferencing

Introduction

With the rapid stride of artificial intelligence & machine learning and businesses integrating these into their products and operations, safeguarding sensitive data and models is a top priority. That’s where Confidential Containers (CoCo) comes into picture. Confidential Containers:

• Provides an extra layer of protection for data in use.
• Helps prevent data leaks.
• Prevents tampering and unauthorized access to sensitive data and models.

By integrating CoCo with model-serving frameworks like KServe¹, businesses can create a secure environment for deploying and managing machine learning models. This integration is critical in strengthening data protection strategies and ensuring that sensitive information stays safe.

Model Inferencing

Model inferencing typically occurs on large-scale cloud infrastructure. The following diagram illustrates how users interact with these deployments.

Importance of Model Protection

Protecting both the model and the data is crucial. The loss of the model leads to a loss of intellectual property (IP), which negatively impacts the organization’s competitive edge and revenue. Additionally, any loss of user data used in conjunction with the model can erode users’ trust, which is a vital asset that, once lost, can be difficult to regain.

Additionally, reputational damage can have long-lasting effects, tarnishing a company’s image in the eyes of both current and potential customers. Ultimately, the loss of a model can diminish a company’s competitive advantage, setting it back in a race where innovation and trustworthiness are key.

Attack Vectors against Model Serving Platforms

Model serving platforms are critical for deploying machine learning solutions at scale. However, they are vulnerable to several common attack vectors. These attack vectors include the following:

• Data or model poisoning: Introducing malicious data to corrupt the model’s learning process.
• Data privacy breaches: Unauthorized access to sensitive data.
• Model theft: Proprietary or fine-tuned models are illicitly copied or stolen.
• Denial-of-service attacks: Overwhelming the system to degrade performance or render it inoperable.

The OWASP Top 10 for LLMs paper² provides a detailed explanation of the different attack vectors.

Among these attack vectors, our focus here is “model theft” as it directly jeopardizes the intellectual property and competitive advantage of organizations.

Traditional Model Protection Mechanisms

Kubernetes offers various mechanisms to harden the cluster in order to limit the access to data and code. Role-Based Access Control (RBAC) is a foundational pillar regulating who can interact with the Kubernetes API and how. Thus ensuring that only authorized personnel have access to sensitive operations. API security mechanisms complements RBAC and acts as gatekeeper, safeguarding the integrity of interactions between services within the cluster. Monitoring, logging, and auditing further augment these defences by providing real-time visibility into the system’s operations, enabling prompt detection and remediation of any suspicious activities.

Additionally, encrypting models at rest ensures that data remains secure even when not in active use, while using Transport Layer Security (TLS) for data in transit between components in the cluster protects sensitive information from interception, maintaining the confidentiality and integrity of data as it moves within the Kubernetes environment. These layered security measures create a robust framework for protecting models against threats, safeguarding the valuable intellectual property and data they encapsulate.

But, is this enough?

Demo: Read Unencrypted Memory

This video showcases how one can read the pod memory when it is run using the default runc³ or kata-containers⁴. But using kata’s confidential compute⁵ support we can avoid exposing the memory to the underlying worker node.

Confidential Containers (CoCo)

The Confidential Containers (CoCo) project aims at integrating confidential computing⁶ into Kubernetes, offering a transformative approach to enhancing data security within containerized applications. By leveraging Trusted Execution Environments (TEEs)⁷ to create secure enclaves for container execution, CoCo ensures that sensitive data and models are processed in a fully isolated and encrypted memory environment. CoCo not only shields the memory of applications hosting the models from unauthorized access but also from privileged administrators who might have access to the underlying infrastructure.

As a result, it adds a critical layer of security, protecting against both external breaches and internal threats. The confidentiality of memory at runtime means that even if the perimeter defenses are compromised, the data and models within these protected containers remain impenetrable, ensuring the integrity and confidentiality of sensitive information crucial for maintaining competitive advantage and user trust.

KServe

KServe¹ is a model inference platform on Kubernetes. By embracing a broad spectrum of model-serving frameworks such as TensorFlow, PyTorch, ONNX, SKLearn, and XGBoost, KServe facilitates a flexible environment for deploying machine learning models. It leverages Custom Resource Definitions (CRDs), controllers, and operators to offer a declarative and uniform interface for model serving, simplifying the operational complexities traditionally associated with such tasks.

Beyond its core functionalities, KServe inherits all the advantageous features of Kubernetes, including high availability (HA), efficient resource utilization through bin-packing, and auto scaling capabilities. These features collectively ensure that KServe can dynamically adapt to changing workloads and demands, guaranteeing both resilience and efficiency in serving machine learning models at scale.

KServe on Confidential Containers (CoCo)

In the diagram below we can see that we are running the containers hosting models in a confidential computing environment using CoCo. Integrating KServe with CoCo offers a transformative approach to bolstering security in model-serving operations. By running model-serving containers within the secure environment provided by CoCo, these containers gain memory protection. This security measure ensures that both the models and the sensitive data they process, including query inputs and inference outputs, are safeguarded against unauthorized access.

Image Source⁸

Such protection extends beyond external threats, offering a shield against potential vulnerabilities posed by infrastructure providers themselves. This layer of security ensures that the entire inference process, from input to output, remains confidential and secure within the protected memory space, thereby enhancing the overall integrity and reliability of model-serving workflows.

Takeaways

Throughout this exploration, we’ve uncovered the pivotal role of Confidential Containers (CoCo) in fortifying data protection, particularly for data in use. CoCo emerges as a comprehensive solution capable of mitigating unauthorized in-memory data access risks. Model-serving frameworks, such as KServe, stand to gain significantly from the enhanced security layer provided by CoCo, ensuring the protection of sensitive data and models throughout their operational life cycle.

However, it’s essential to recognize that not all components must operate within CoCo’s protected environment. A strategic approach involves identifying critical areas where models and data are most vulnerable to unauthorized access and focusing CoCo’s protective measures on these segments. This selective application ensures efficient resource utilization while maximizing data security and integrity.

Further

In the next blog we will see how to deploy KServe on Confidential Containers for memory protection.

Note: This blog is a transcription of the talk we gave at Kubecon EU 2024. You can find the slides on Sched⁹ and the talk recording on YouTube¹⁰.

Building Trust into OS images for Confidential Containers

Containers and OS Images

Confidential Containers using Kata-Containers are launched in a Confidential Virtual Machine (CVM). Those CVMs require a minimal Linux system which will run in our Trusted Execution Environment (TEE) and host the agent side of Kata-Containers (including various auxiliary attestation tools) to launch containers and facilitate secure key releases for a confidential Pod. Integrity of the workload is one of the key pillars for Confidential Computing. Consequently, this implies we also must trust the infrastructure components that host containers on a confidential guest VM, specifically: firmware, rootfs, kernel and kernel cmdline.

For a TEE there are various options to establish this kind of trust. Which option will be used depends on the capabilities and specifics of a TEE. All of them will include various degrees of “measurements” (that is: cryptographic hashes for a blob of data and/or code) to reach the same goal: providing a verifiable statement about the integrity of the OS image. We’ll discuss three viable options; those are not exhaustive

Initial ramdisk

We can opt to not use a rootfs and bundle the required userland components into Linux’ initial ramdisk (initrd), which is loaded by the kernel. Outside a CoCo scenario this facility is used to provide a boot stage in which kernel drivers can be loaded on-demand from a memory-backed (compressed) volume, not having to bundle device drivers for various hardware statically in each vendor kernel. For CoCo VMs, this kind of flexibility is not really required: we do know beforehand the virtualized hardware that our CVM is configured with, and it will require only a limited set of drivers. Due to its static nature, relying solely on an initrd would be impractical for many workloads. For CoCo however, this is a viable option, since the dynamic aspect of its workload is mostly deferred to the container execution. This means we can have the kernel launch a kata-agent as PID 1 directly from an initrd.

This option is appealing for certain CoCo deployments. If we have a Trusted Execution Environment (TEE) that will produce a launch-measurement of the initial RAM state of a CVM, we can use this measurement to gain confidence that our os image is genuine. We can calculate the expected value of a given launch measurement offline and then verify during remote attestation that the actual launch measurement matches our expected value.

Calculating a launch measurement

An expected SEV-SNP launch measurement for Linux direct boot with Qemu can be calculated using trusted artifacts (firmware, kernel & initrd) and a few platform parameters. Please note that the respective kernel/fw components and tools are still being actively developed. The AMDESE/AMDSEV repository provides instructions and pointers to a working set of revisions.

$ sev-snp-measure \
	--mode snp \
	--vcpus=1 \
	--vcpu-type=EPYC-Milan-v1 \
	--kernel=vmlinuz \
	--initrd=initrd.img \
	--append="console=ttyS0" \
	--ovmf OVMF.fd
20f28c1e85c4250c2c061d1997cfc815185cefe756c74b37ea1c81eb8f2e0e3c5c43e58d65e0e792ed2bd04a0720f970

DM Verity

The infrastructure components might outgrow a reasonably sized initrd. We want to limit the initrd to a smaller size, to not spend too much of the CVM’s RAM, but leave as much as possible for the container payload. In the worst case we’ll have to spend ~2x the size of the initrd (we also need to keep the compressed initrd in RAM). With an increasing amount of supported TEEs the infrastructure components will inevitably grow, since they need to support various attestation schemes and hardware. Some of those schemes might also have a larger dependency tree, which is unreasonable to statically link or bundle into an initrd. There might be compliance requirements which mandate the use of certain cryptographic libraries that must not be statically compiled. Those considerations might nudge us to a more traditional Linux setup of kernel, initrd and rootfs.

A rootfs can comfortably host the infrastructure components and we can still package support for all kinds of TEE in a single OS image artifact. However, the dependencies for a given TEE can now be loaded dynamically into RAM. For CVMs there is a restriction when it comes to how an OS image is handled: We must prevent the CVM host, storage provider or anyone else outside the TEE from compromising the image to uphold the integrity of a CoCo workload. dm-verity is a kernel technology that prevents changes to a read-only filesystem during runtime. Block-level access is guarded by a hash tree and on unexpected block data the kernel will panic. This protection scheme requires a root hash that needs to be provided when the verity-protected filesystem is mounted. We can provide this root hash through the kernel cmdline or bake it into the initrd. In any case, the TEE has to include the root hash in a launch measurement to provide verifiable integrity guarantees.

Creating a verity volume

DM-Verity volumes feature a hash tree and a root hash in addition to the actual data. The hash tree can be stored on disk next to the verity volume or as a local file. We’ll store the hash-tree as file for brevity and write a string CoCo into a file /coco on the formatted volume:

$ dd if=/dev/zero of=rootfs.raw bs=1M count=100
$ DEVICE="$(sudo losetup --show -f rootfs.raw)"
$ sudo cfdisk "$DEVICE"
# create 1 partition
$ sudo mkfs.ext4 "$DEVICE"
...
$ sudo mount "$DEVICE" /mnt
$ echo "CoCo" | sudo tee /mnt/coco
CoCo
$ sudo umount /mnt
$ sudo veritysetup format "$DEVICE" ./hash-tree
VERITY header information for ./hash-tree
UUID:                   91bbc990-f0df-48c0-b8f0-1b996cf0c3cf
Hash type:              1
Data blocks:            25600
Data block size:        4096
Hash block size:        4096
Hash algorithm:         sha256
Salt:                   cef7ea72e3487f4f8d26df8731df561f64e03236fa494dc0ae87fe0f07a4825b
Root hash:              ad86ff8492be2ee204cb54d70c84412c2dc89cefd34e263184f4e00295a412f3
$ export ROOT_HASH=ad86ff8492be2ee204cb54d70c84412c2dc89cefd34e263184f4e00295a412f3

Corrupting the image

Now we toggle a bit on the raw image (CoCo => DoCo in /coco). If the image is attached as a block device via dm-verity, there will be IO errors and respective entries in the kernel log, once we attempt to read the file.

$ hexdump -C rootfs.raw | grep CoCo
06000000  43 6f 43 6f 0a 00 00 00  00 00 00 00 00 00 00 00  |CoCo............|
$ printf '\x44' | dd of=rootfs.raw bs=1 seek="$((16#06000000))" count=1 conv=notrunc
$ hexdump -C rootfs.raw | grep DoCo
06000000  44 6f 43 6f 0a 00 00 00  00 00 00 00 00 00 00 00  |DoCo............|
$ sudo veritysetup open "$DEVICE" rootfs ./hash-tree "$ROOT_HASH"
$ sudo mount /dev/mapper/rootfs /mnt
$ cat /mnt/coco
cat: /mnt/coco: Input/output error
$ dmesg | tail -1
[194754.361797] device-mapper: verity: 7:0: data block 24576 is corrupted

vTPM

There are setups in which a launch measurement of the TEEs will not cover the kernel and/or initrd. An example of such a TEE is Azure’s Confidential VM offering (L1 VMs provided by hypervisor running on a physical host). Those CVMs can host Confidential Containers in a CoCo Peerpod setup. The hardware evidence, which is attesting encrypted RAM and CPU registers is exclusively fetched during an early boot phase. Only in later stages the kernel and initrd are loaded from an OS image and hence the launch measurement will not cover the CoCo infrastructure components yet. To still be able to provide an integrity guarantee such a TEE can defer measurements of the later boot stages to a virtual TPM device (vTPM).

To isolate it from the host a confidential vTPM is provisioned within the TEE during early boot and cryptographically linked to the TEE’s hardware evidence. To further secure secrets like private keys from the guest OS, the provisioning is performed at a certain privilege level preventing direct access and manipulation by the guest OS which is running at a lesser privilege level.

TPM is a mature technology, deployed in a lot of hardware to protect operating systems and workloads from being compromised. It’s seeing increased adoption and support in the Linux kernel and userland. A TPM device has multiple Platform Configuration Registers (PCR). Those can hold measurements and they can be “extended” with additional measurements in a one-way function to create a comprehensive, replayable log of events that occur during the boot process. “Measured Boot” is a procedure in which each boot step measures the subsequent step into a specific PCR. As a whole this represents a verifiable state of the system, much like an initial launch measurement, however with more granularity.

Image building

Modern OS Image build tools for Linux like systemd’s mkosi make it trivial to build OS images with dm-verity protection enabled, along with Unified Kernel Images (UKI) which bundles kernel, initrd and kernel cmdline into conveniently measurable artifacts. A modern distribution packaging recent systemd (v253+) revisions like Fedora (38+) will perform the required TPM measurements.

Creating reference values

To retrieve the expected measurements, for a dm-verity protected OS image, we can boot the resulting image in a trusted environment locally. The swtpm project is a great option to provide the virtual machine with a vTPM.

$ swtpm socket \
	--tpmstate dir=/tmp/vtpm \
	--ctrl type=unixio,path=/tmp/vtpm/swtpm.sock \
	--tpm2 \
	--log level=20

We retrieve VM firmware from debian’s repository and attach the vTPM socket as character device:

# retrieve vm firmware from debian's repo
$ wget http://security.debian.org/debian-security/pool/updates/main/e/edk2/ovmf_2022.11-6+deb12u1_all.deb
$ mkdir fw
$ dpkg-deb -x ovmf_2022.11-6+deb12u1_all.deb fw/
$ cp fw/usr/share/OVMF/OVMF_*.fd .
$ OS_IMAGE=image.raw
$ qemu-system-x86_64 \
	-machine type=q35,accel=kvm,smm=off \
	-m 1024 \
	-drive file=./OVMF_CODE.fd,format=raw,if=pflash \
	-drive file=./OVMF_VARS.fd,format=raw,if=pflash \
	-drive "file=${OS_IMAGE},format=raw" \
	-chardev socket,id=chrtpm,path=/tmp/vtpm/swtpm.sock \
	-tpmdev emulator,id=tpm0,chardev=chrtpm \
	-device tpm-tis,tpmdev=tpm0 \
	-nographic

Comparing PCRs

Once logged into the VM we can retrieve the relevant measurements in the form of PCRs (the package tpm2_tools needs to be available):

$ tpm2_pcrread sha256:0,1,2,3,4,5,6,7,8,9,10,11
  sha256:
	0 : 0x61E3B90D0862D052BF6C802E0FD2A44A671A37FE2EB67368D89CB56E5D23014E
	1 : 0x33D454DFCF5E46C0B7AFD332272E72ADC3D1A86CCAE25AA98DD475C9FCA36CFC
	2 : 0x7842C772A64365B48AC733EDEE9B131DF5F0E71EA95074F80E32450995C5773D
	3 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
	4 : 0x22B156BE656EED7542AB03CC76DCC8A82F2A31044B5F17B3B8A388CB8DB37850
	5 : 0x3F72C8A7A38564991898859F725D12E5BE64CBD26265BC8F5E39CBE1101EBD49
	6 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
	7 : 0x65CAF8DD1E0EA7A6347B635D2B379C93B9A1351EDC2AFC3ECDA700E534EB3068
	8 : 0x0000000000000000000000000000000000000000000000000000000000000000
	9 : 0x8E74577DC5814F2EBF094988CB2E789F1D637B4D43930F3714500F9E2E65615D
	10: 0x961D21A6CB38D377F951748BA7B8DD05A2E1BA6C712BB34EF7A39C5862721F1E
	11: 0x9DBA7A9D3C5200B0E526112151BBD23D77006CBFCF290CFA6249601CA9812608

If we boot the same image on a Confidential VM in Azure’s cloud, we’ll see different measurements. This is expected since the early boot stack does not match our reference setup:

$ tpm2_pcrread sha256:0,1,2,3,4,5,6,7,8,9,10,11
  sha256:
	0 : 0x782B20B10F55CC46E2142CC2145D548698073E5BEB82752C8D7F9279F0D8A273
	1 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
	2 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
	3 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
	4 : 0xC7BB081502F18392EB5837951A9BA48E9DB23F91DE39A9AF8B2B29C333D71EA0
	5 : 0x0358DC1195BBDD59E3C556A452E292A6E7ECF11408BE7DAEC6776E678BEBEC23
	6 : 0x531086506EADC75D0E540F516D68E03095E5700FE8F1BD0F840025B07A3AB4F7
	7 : 0x64CDD65955B69C5ADD78577E32BFE52DDF9ADBF240977AEA39703908F4F6D8BA
	8 : 0x0000000000000000000000000000000000000000000000000000000000000000
	9 : 0x8E74577DC5814F2EBF094988CB2E789F1D637B4D43930F3714500F9E2E65615D
	10: 0x5A7ACDE0EF2AB221551CB24CCFDB7AE959047E3C0E0C39427D329992A9C7FDDF
	11: 0x9DBA7A9D3C5200B0E526112151BBD23D77006CBFCF290CFA6249601CA9812608

We can identify the common PCRs between the measurements in a cloud VM and those that we gathered in our reference setup. Those are good candidates to include them as reference values in a relying party against which a TEE’s evidence can be verified.

$ grep -F -x -f pcr_reference.txt pcr_cloud.txt
	3 : 0x3D458CFE55CC03EA1F443F1562BEEC8DF51C75E14A9FCF9A7234A13F198E7969
	8 : 0x0000000000000000000000000000000000000000000000000000000000000000
	9 : 0x8E74577DC5814F2EBF094988CB2E789F1D637B4D43930F3714500F9E2E65615D
	11: 0x9DBA7A9D3C5200B0E526112151BBD23D77006CBFCF290CFA6249601CA9812608

The UAPI Group’s TPM PCR Registry for Linux and systemd specifies PCR11 as a container for UKI measurements, covering kernel, initrd and kernel cmdline. Further registers that might be worth considering would be PCR4 (shim + UKI) or PCR7 (Secure Boot state).

Conclusion and outlook

We have looked at three different ways of building trust into OS host images for Confidential Containers. The intention was to illustrate how a chain of trust can be established using concrete examples and tools. The scenarios and technologies haven’t been covered comprehensively, each of those would be worth their own in-depth article.

Finally we have so far only covered the (mostly static) steps and components that provide a sandbox for confidential containers. Asserting integrity for containers themselves is a unique challenge for CoCo. There are a lot of dynamic aspects to consider in a realistic container deployment. Future articles might provide insights into how this can be achieved.

Thanks to Pradipta Banerjee, Iago López Galeiras & Tobin Feldman-Fitzthum for reviewing this post!

Introduction to Confidential Containers (CoCo)

Confidential Containers (CoCo) is an innovative sandbox project under the Cloud Native Computing Foundation (CNCF), revolutionizing cloud-native confidential computing by leveraging diverse hardware platforms and cutting-edge technologies.

The CoCo project builds on existing and emerging hardware security technologies such as Intel SGX, Intel TDX, AMD SEV-SNP and IBM Z Secure Execution, in combination with new software frameworks to protect data in use. The project brings together software and hardware companies including Alibaba-cloud, AMD, ARM, Edgeless Systems, IBM, Intel, Microsoft, Nvidia, Red Hat, Rivos, etc.

Motivation

At the core of a confidential computing solution lies Trusted Execution Environments (TEEs), and it is this foundational idea that propelled the inception of the CoCo project.

TEEs represent isolated environments endowed with heightened security, a shield crafted by confidential computing (CC) capable hardware. This security fortress stands guard, ensuring that applications and data remain impervious to unauthorized access or tampering during their active use.

The driving force behind CoCo is the seamless integration of TEE infrastructure into the realm of cloud-native computing. By bridging the gap between TEEs and the cloud-native world, the project strives to bring enhanced security to the forefront of modern computing practices.

The overarching goal of CoCo is ambitious yet clear: standardize confidential computing at the container level and simplify its integration into Kubernetes.

The aim is to empower Kubernetes users to deploy confidential container workloads effortlessly, using familiar workflows and tools. CoCo envisions a future where Kubernetes users can embrace the benefits of confidential computing without the need for extensive knowledge of the underlying technologies, making security an integral and accessible aspect of their everyday operations.

Mechanics

CoCo helps in deploying your workload that extends beyond the confines of your own infrastructure. Whether it’s a cloud provider’s domain, a separate division within your organization, or even an external entity, CoCo empowers you to confidently entrust your workload to diverse hands.

This capability hinges on a fundamental approach: encrypting your workload’s memory and fortifying other essential low-level resources at the hardware level. This memory protection ensures that, regardless of the hosting environment, your data remains shielded, and unauthorized access is thwarted.

A key aspect of CoCo’s mechanics lies in the use of cryptography-based proofs which involve employing cryptographic techniques to create verifiable evidence, such as signatures or hashes, ensuring the integrity of your software. These serve a dual purpose: validating that your software runs untampered and, conversely, preventing the execution of your workload if any unauthorized alterations are detected.

In essence, CoCo employs cryptographic mechanisms to provide assurance, creating a secure foundation that allows your software to operate with integrity across varied and potentially untrusted hosting environments.

Foundational Principles

The project puts a strong emphasis on delivering practical cloud-native solution:

• Simplicity: CoCo places a premium on simplicity, employing a dedicated Kubernetes operator for deployment and configuration. This strategic choice aims to maximize accessibility by abstracting away much of the hardware-dependent intricacies, ensuring a user-friendly experience.

• Stability: Supporting continuous integration (CI) for the key workflows of the release.

• Use case driven development: CoCo adopts a use case-driven development approach, rallying the community around a select set of key use cases. Rather than a feature-centric model, this strategy ensures that development efforts are purposeful, with a spotlight on supporting essential use cases. This pragmatic approach aligns the project with real-world needs, making CoCo a solution crafted for practical cloud-native scenarios.

Community

Discover the vibrant CoCo community and explore ways to actively engage with the project by visiting our dedicated community page. We welcome and actively seek your thoughts, feedback, and potential contributions. Join us in shaping the future of confidential containers and explore collaborative opportunities to integrate CoCo with other cloud-native projects. Your participation is not just encouraged; it’s integral to the evolution and success of this open-source initiative. Visit the community page now to be a part of the conversation and contribute to the advancement of confidential computing in the cloud-native ecosystem.

See our CoCo community meeting notes for details on the weekly meetings, recordings, slack channels and more.