Resources

• 1: Resource Backends
• 2: Azure Key Vault Integration

kbs:///<repository>/<type>/<tag>`

kbs-client --url <kbs-url> config \
    --auth-private-key <admin-private-key> set-resource \
    --path <resource-path> --resource-file <path-to-resource-file>

cat "$KEY_FILE" | kubectl exec -i deploy/trustee-deployment -n trustee-operator-system -- tee "/opt/confidential-containers/kbs/repository/${KEY_PATH}" > /dev/null

kubectl create secret generic kbsres1 --from-literal key1=res1val1 --from-literal key2=res1val2 -n kbs-operator-system

1 - Resource Backends

Resource Storage Backend

KBS stores confidential resources through a StorageBackend abstraction specified by a Rust trait. The StorageBackend interface can be implemented for different storage backends like e.g. databases or local file systems.

The KBS config file defines which resource backend KBS will use. The default is the local file system (LocalFs).

Local File System Backend

With the local file system backend default implementation, each resource file maps to a KBS resource URL. The file path to URL conversion scheme is defined below:

The KBS root file system resource path is specified in the KBS config file as well, and the default value is /opt/confidential-containers/kbs/repository.

Aliyun KMS

Alibaba Cloud KMS(a.k.a Aliyun KMS) can also work as the KBS resource storage backend. In this mode, resources will be stored with generic secrets in a KMS instance. One KBS can be configured with a specified KMS instance in repository_config field of KBS launch config. These materials can be found in KMS instance’s AAP. When being accessed, a resource URI of kbs:///repo/type/tag will be translated into the generic secret with name tag. Hinting that repo/type field will be ignored.

Pkcs11

The Pkcs11 backend uses Pkcs11 to store plaintext resources in an HSM. Pkcs11 is a broad specification supporting many cryptographic operations. Here we make use only of a small subset of these features. Often with Pkcs11 an HSM is used to wrap and unwrap keys or store wrapped keys. Here we do something simpler. Since the KBS expects resources to be in plaintext, we store these resources in the HSM as secret keys of the generic secret type. This storage backend will provision resource to the HSM in the expected way when a user uploads a resource to the KBS. The user must simply specify the location of an initialized HSM slot. Keys can also be provisioned to the HSM separately but they must have the expected attributes.

The Pkcs11 backend is configured with the following values.

• module The module path should point to a binary implementing Pkcs11 for the HSM that you want to use. For example, if you are using SoftHSM, you might set the module path to /usr/local/lib/softhsm/libsofthsm2.so.
• slot_index The slot index points to the slot in your HSM where the secrets will be stored. The slot must be initialized before starting the KBS. No slot_index is set, the first slot will be used.
• pin The user password for authenticating a session with the above slot.

2 - Azure Key Vault Integration

Premise

AKS

We assume an AKS cluster configured with Workload Identity and Key Vault Secrets Provider. The former provides a KBS pod with the privileges to access an Azure Key Vault (AKV) instance. The latter is an implementation of Kubernetes’ Secret Store CSI Driver, mapping secrets from external key vaults into pods. The guides below provide instructions on how to configure a cluster accordingly:

• Use the Azure Key Vault provider for Secrets Store CSI Driver in an Azure Kubernetes Service (AKS) cluster
• Use Microsoft Entra Workload ID with Azure Kubernetes Service (AKS)

AKV

There should be an AKV instance that has been configured with role based access control (RBAC), containing two secrets named coco_one coco_two for the purpose of the example. Find out how to configure your instance for RBAC in the guide below.

Provide access to Key Vault keys, certificates, and secrets with an Azure role-based access control

Note: You might have to toggle between Access Policy and RBAC modes to create your secrets on the CLI or via the Portal if your user doesn’t have the necessary role assignments.

CoCo

While the steps describe a deployment of KBS, the configuration of a Confidential Containers environment is out of scope for this document. CoCo should be configured with KBS as a Key Broker Client (KBC) and the resulting KBS deployment should be available and configured for confidential pods.

Azure environment

Configure your Resource group, Subscription and AKS cluster name. Adjust accordingly:

export SUBSCRIPTION_ID="$(az account show --query id -o tsv)"
export RESOURCE_GROUP=my-group
export KEYVAULT_NAME=kbs-secrets
export CLUSTER_NAME=coco

Instructions

Create Identity

Create a User managed identity for KBS:

az identity create --name kbs -g "$RESOURCE_GROUP"
export KBS_CLIENT_ID="$(az identity show -g "$RESOURCE_GROUP" --name kbs --query clientId -o tsv)"
export KBS_TENANT_ID=$(az aks show --name "$CLUSTER_NAME" --resource-group "$RESOURCE_GROUP" --query identity.tenantId -o tsv)

Assign a role to access secrets:

export KEYVAULT_SCOPE=$(az keyvault show --name "$KEYVAULT_NAME" --query id -o tsv)
az role assignment create --role "Key Vault Administrator" --assignee "$KBS_CLIENT_ID" --scope "$KEYVAULT_SCOPE"

Namespace

By default KBS is deployed into a coco-tenant Namespace:

export NAMESPACE=coco-tenant
kubectl create namespace $NAMESPACE

KBS identity and Service Account

Workload Identity provides individual pods with IAM privileges to access Azure infrastructure resources. An azure identity is bridged to a Service Account using OIDC and Federated Credentials. Those are scoped to a Namespace, we assume we deploy the Service Account and KBS into the default Namespace, adjust accordingly if necessary.

export AKS_OIDC_ISSUER="$(az aks show --resource-group "$RESOURCE_GROUP" --name "$CLUSTER_NAME" --query "oidcIssuerProfile.issuerUrl" -o tsv)"
az identity federated-credential create \
 --name kbsfederatedidentity \
 --identity-name kbs \
 --resource-group "$RESOURCE_GROUP" \
 --issuer "$AKS_OIDC_ISSUER" \
 --subject "system:serviceaccount:${NAMESPACE}:kbs"

Create a Service Account object and annotate it with the identity’s client id.

cat <<EOF> service-account.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: ${KBS_CLIENT_ID}
  name: kbs
  namespace: ${NAMESPACE}
EOF
kubectl apply -f service-account.yaml

Secret Provider Class

A Secret Provider Class specifies a set of secrets that should be made available to k8s workloads.

cat <<EOF> secret-provider-class.yaml
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: ${KEYVAULT_NAME}
  namespace: ${NAMESPACE}
spec:
  provider: azure
  parameters:
    usePodIdentity: "false"
    clientID: ${KBS_CLIENT_ID}
    keyvaultName: ${KEYVAULT_NAME}
    objects: |
      array:
      - |
        objectName: coco_one
        objectType: secret
      - |
        objectName: coco_two
        objectType: secret
    tenantId: ${KBS_TENANT_ID}
EOF
kubectl create -f secret-provider-class.yaml

Deploy KBS

The default KBS deployment needs to be extended with label annotations and CSI volume. The secrets are mounted into the storage hierarchy default/akv.

git clone https://github.com/confidential-containers/kbs.git
cd kbs
git checkout v0.8.2
cd kbs/config/kubernetes
mkdir akv
cat <<EOF> akv/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: coco-tenant

resources:
- ../base

patches:
- path: patch.yaml
  target:
    group: apps
    kind: Deployment
    name: kbs
    version: v1
EOF
cat <<EOF> akv/patch.yaml
- op: add
  path: /spec/template/metadata/labels/azure.workload.identity~1use
  value: "true"
- op: add
  path: /spec/template/spec/serviceAccountName
  value: kbs
- op: add
  path: /spec/template/spec/containers/0/volumeMounts/-
  value:
    name: secrets
    mountPath: /opt/confidential-containers/kbs/repository/default/akv
    readOnly: true
- op: add
  path: /spec/template/spec/volumes/-
  value:
    name: secrets
    csi:
      driver: secrets-store.csi.k8s.io
      readOnly: true
      volumeAttributes:
        secretProviderClass: ${KEYVAULT_NAME}
EOF
kubectl apply -k akv/

Test

The KBS pod should be running, the pod events should give indication of possible errors. From a confidential pod the AKV secrets should be retrievable via Confidential Data Hub:

$ kubectl exec -it deploy/nginx-coco -- curl http://127.0.0.1:8006/cdh/resource/default/akv/coco_one
a secret