Kubernetes

The Helm chart deploys the service on any state store. With the Kubernetes store it needs nothing but the cluster: no database, and with workload identity on AKS, no secret of its own.

helm install tresor oci://ghcr.io/hugr-lab/charts/tresor-server -n tresor --create-namespace -f values.yaml

Kubernetes 1.30 or later. The chart's values.yaml documents every value; its README, the main ones.

The Kubernetes store, the KEK in Key Vault (AKS)

serviceAccount: {name: tresor}
workloadIdentity: {enabled: true, clientId: <the managed identity's client id>}
config:
  public_url: https://tresor.example.com
  state: {kind: kubernetes}
  keys: {kind: azurekeyvault, key: https://corp-kv.vault.azure.net/keys/tresor-kek}
  azure: {identity: workload}
  issuers:
    - {issuer: https://login.microsoftonline.com/<tenant>/v2.0, audience: <api client id>, roles_claim: roles}
  policy: {admins: [role:secrets_admin]}
ingress:
  enabled: true
  className: nginx            # unless the cluster has a default class
  hosts: [{host: tresor.example.com}]
  tls: [{secretName: tresor-tls, hosts: [tresor.example.com]}]   # cert-manager, or a TLS Secret of yours

On Azure:

• The cluster needs --enable-oidc-issuer --enable-workload-identity.

• The KEK: an RSA key with the operations wrapKey, unwrapKey and sign (sign makes the root, see Encryption).

• A user-assigned identity, federated with the release's ServiceAccount:

  az identity create -g <rg> -n tresor
  az identity federated-credential create -g <rg> --identity-name tresor -n tresor-sa \
    --issuer "$(az aks show -g <rg> -n <cluster> --query oidcIssuerProfile.issuerUrl -o tsv)" \
    --subject system:serviceaccount:<namespace>:<serviceAccount.name> --audiences api://AzureADTokenExchange

• Its rights on the KEK: get, wrap, unwrap and sign, on the one key. No built-in role gives exactly that, so make a custom role:

  az role definition create --role-definition '{
    "Name": "tresor KEK user", "Actions": [],
    "DataActions": ["Microsoft.KeyVault/vaults/keys/read", "Microsoft.KeyVault/vaults/keys/wrap/action",
                    "Microsoft.KeyVault/vaults/keys/unwrap/action", "Microsoft.KeyVault/vaults/keys/sign/action"],
    "AssignableScopes": ["/subscriptions/<subscription>/resourceGroups/<rg>"]}'
  az role assignment create --assignee-object-id <the identity's principal id> --assignee-principal-type ServicePrincipal \
    --role "tresor KEK user" --scope "<the vault's id>/keys/<key>"

• For ref+azkv: config.material.azkv.allow names the vault and the secret-name prefixes. The identity needs Key Vault Secrets User on those secrets.

• A private CA, such as an IdP's: mount it and set SSL_CERT_DIR to its directory. That adds it to the system's roots. SSL_CERT_FILE would replace them, and Key Vault's TLS would fail.

scripts/dev/aks_live.sh does all of this for a live check.

The service stops at start when the pod has no identity injected, and names the label and the annotation to set.

The store

The store is custom resources in the chart's namespace - the release's, or namespaceOverride (see State stores):

• several replicas, compare-and-set through resourceVersion;
• a MAC over everything not sealed: a resource changed by hand is refused;
• the admission policy (cluster-scoped, in the chart): only the service's ServiceAccount writes. The garbage collector may delete minted tokens; the namespace controller deletes only while the namespace is being deleted. It stops the one thing the MAC does not: an older copy put back.

The CRDs

• Helm installs crds/ on the first install, whatever the store, and never upgrades them.
• On another store, install with --skip-crds.
• On the Kubernetes store without cluster rights: a cluster admin applies crds/ first. Install with --skip-crds and admissionPolicy.enabled: false (the policy is cluster-scoped too).
• An upgrade that changes the CRDs says so: apply the new chart's first - helm pull oci://ghcr.io/hugr-lab/charts/tresor-server --version <version> --untar && kubectl apply --server-side -f tresor-server/crds/. Variables (spec 004) brought TresorVariable: an install from before needs it, or the service does not start.
• One installation per namespace: a second release's admission policy would lock the first out.

Kubernetes Secrets as material

config:
  material:
    k8s:
      allow:
        - namespace: data-team
          prefixes: [duckdb-]

• A secret's parameter or a variable can name ref+k8s://data-team/duckdb-lake/secret (see References).
• The chart makes a Role per namespace listed, with get on Secrets.
• The service's own namespace is refused: its credentials are there.

OpenBao or Vault

The KEK in Transit, ref+vault and the exchange's key in Vault, with the chart's vaultToken: see OpenBao and Vault.

Other stores

• PostgreSQL or SQL Server: config.state as on any host.
  • The password can come from a Secret: auth: password with state.password_ref: ref+k8s://db/tresor-pg/password. The chart grants get on that one Secret.
  • Or from workload identity, with auth: entra.
• SQLite: not recommended on a cluster - one replica on a ReadWriteOnce claim, and every command needs the service scaled to 0 first (kubectl scale deploy/<fullname> --replicas=0; its volume is the serving pod's, and mac and reseal need its lease). A cluster can run PostgreSQL or SQL Server: use one.

The commands as jobs

The image has no shell, and nobody logs into the pod. Each command runs as a Job with the service's own ServiceAccount, configuration, volumes and environment (spec 019). The chart makes a suspended CronJob per command (<fullname> is the release's name and the chart's, as the chart's NOTES print it); start one by hand:

kubectl -n tresor create job --from=cronjob/<fullname>-reseal reseal-$(date +%Y%m%d%H%M)
kubectl -n tresor logs -f job/reseal-…        # its log: counts and names, never a value

CronJob	Runs
<fullname>-reseal	reseal -retire
<fullname>-reseal-rotate	reseal -rotate -retire (a data key leaked)
<fullname>-rewrap	rewrap
<fullname>-refs	refs -resolve
<fullname>-mac	mac (the SQL stores)

• A long fullname is cut so that a CronJob's name stays within 52 characters.
• maintenance.reseal.schedule (a cron, e.g. 0 3 1 * *) runs reseal -retire on that schedule (not on SQLite). A job started by hand may overlap it: every change is compare-and-set, so both finish correctly.
• maintenance.refsBeforeUpgrade: true runs refs -resolve with the new configuration before an upgrade (a pre-upgrade hook), when the image stays - a configuration change; a reference it would strand stops the upgrade. An upgrade of the image runs none (the new version's database is migrated by its pods): change the configuration in an upgrade of its own. Helm only - helm template and Argo CD render no hook. The hook runs with the release's current ServiceAccount and RBAC: change those in an upgrade of their own too.
• The jobs' pods carry podLabels (a policy that admits the service admits them) and maintenance.podAnnotations, not podAnnotations - a mesh's sidecar would keep a Job from completing.
• The right to create Jobs in the namespace is the KEK's: whoever has it acts as the service.

The cluster's own accounts

A Job of the cluster - an install Job, say - logs in with its ServiceAccount's projected token when the service trusts the cluster's ServiceAccount issuer (spec 026). Outside Azure there is no identity provider account for it; on AKS a federated managed identity is the other way.

clusterIssuer: {enabled: true}           # the service's own projected token and the cluster's CA, mounted
config:
  issuers:
    - issuer: https://kubernetes.default.svc.cluster.local   # the cluster's: kubectl get --raw /.well-known/openid-configuration
      audience: duckdb-secrets                               # the Job's projected token's audience
      ca_file: /var/run/tresor/cluster/ca.crt
      discovery_token_file: /var/run/tresor/cluster/token
      service: {claim: kubernetes.io, client_claim: sub}     # a ServiceAccount is a service: client:system:serviceaccount:<ns>:<name>
  policy:
    admins: ["client:system:serviceaccount:hugr:hugr-init"]

• The issuer's discovery: the API server serves it to authenticated accounts. Bind the service's account to system:service-account-issuer-discovery (a ClusterRoleBinding). discovery_token_file needs ca_file, and then the issuer's hosts are trusted by the cluster's CA only: the token - the service's Kubernetes credential - never reaches a host a public CA certifies.
• A public issuer (AKS, EKS, GKE with their OIDC issuers enabled): its discovery is public - set neither ca_file nor discovery_token_file.
• The Job's token: a projected serviceAccountToken with audience: duckdb-secrets, sent as a bearer token. Verified offline (the issuer's keys): a deleted account's token holds until its exp - keep it short.
• Who: an administrator is all-or-nothing - list the one account, in its own namespace: whoever creates pods there acts as it. With another issuer beside the cluster's, name it by its subject - subject:<the cluster's issuer>|system:serviceaccount:hugr:hugr-init - a client: name is not tied to an issuer, and another issuer's client of that name would pass for it.
• The CA is read once: a rotation of the cluster's CA reaches the service at its restart.

Installed beside others: namespaceOverride, configTemplate

An umbrella chart installs tresor-server in a namespace of its own: namespaceOverride puts every object of the chart there, whatever the release's namespace. The namespace must exist (Helm creates only the release's). It is a trust boundary - whoever creates pods in it can act as its accounts, the service's included.

configTemplate: true renders config through tpl first, so an umbrella chart's global values reach tresor's settings without being written twice:

configTemplate: true
config:
  state:
    dsn: "host={{ .Values.global.database.host }} dbname=tresor sslmode=verify-full"
  policy:
    admins: ["subject:{{ .Values.global.clusterIssuer }}|system:serviceaccount:{{ .Release.Namespace }}:hugr-init"]

Off by default: with it, a literal {{ in config is written {{ "{{" }}. Only the values are rendered: the service's own syntax (${…}, mint+…) lives in its secrets, not in config.

Hardening

• With tls.offload the pod serves plain HTTP. Set networkPolicy.from to the ingress controller.
• Whoever can create pods or Jobs in the service's namespace, mint a token for its ServiceAccount or impersonate it can act as the service. Keep these rights to the platform's operators.
• A restore (Velero) writes as another account. Remove the admission policy's binding for its duration.