OpenBao and HashiCorp Vault

The service uses OpenBao or HashiCorp Vault (spec 007) for three things, each optional:

• the KEK in Transit (keys.kind: vault), see Encryption;
• references to KV v2 (ref+vault://), see References;
• the exchange's assertion signed in Transit (client_auth: vault), see Token exchange.

The same API serves both. CI tests against OpenBao 2.4 and Vault 2.1.

Together with the Kubernetes store and ZITADEL, this is a stack with no cloud at all: no static secret of the service's own, no key in a file.

The login

The service logs in with no static secret:

vault.auth.method	The token it logs in with	Set-up in Vault
kubernetes	a ServiceAccount token: the chart's projected one (vaultToken), or the pod's own	the Kubernetes auth method
jwt	a projected ServiceAccount token (jwt_file)	the JWT auth method, trusting the cluster's issuer
token_file	a token a Vault Agent writes and renews	the Agent's own login

• kubernetes suits Vault in the cluster: it asks the cluster's API to review the token, as its own ServiceAccount.
  • A Vault outside the cluster needs a reviewer of its own: token_reviewer_jwt (a ServiceAccount bound to system:auth-delegator) and kubernetes_ca_cert. Without it Vault reviews with the service's own token, which vaultToken binds to Vault's audience: the review fails. Prefer jwt there.
• jwt suits a Vault outside, which checks the token against the cluster's OIDC issuer and never calls the cluster's API.

Kubernetes auth

bao auth enable kubernetes
bao write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc
bao write auth/kubernetes/role/tresor \
  bound_service_account_names=tresor-tresor-server bound_service_account_namespaces=tresor \
  audience=vault token_policies=tresor token_ttl=1h token_type=batch

• With Vault in the cluster, its own ServiceAccount reviews the tokens: bind it to system:auth-delegator.
• audience=vault is the chart's vaultToken.audience. The projected token is then good for Vault only.

JWT auth

bao auth enable jwt
bao write auth/jwt/config oidc_discovery_url=https://<the cluster's issuer>
bao write auth/jwt/role/tresor role_type=jwt user_claim=sub \
  bound_audiences=vault bound_subject=system:serviceaccount:tresor:tresor-tresor-server \
  token_policies=tresor token_ttl=1h token_type=batch

• Vault must reach the issuer's discovery and JWKS. On a self-managed cluster that is the API server: let anyone read them (bind system:service-account-issuer-discovery to system:unauthenticated), and give oidc_discovery_ca_pem for its CA. Or give the keys themselves: jwt_validation_pubkeys.

The policy

One policy, naming only what the service uses:

# the KEK
path "transit/keys/tresor-kek"           { capabilities = ["read"] }
path "transit/encrypt/tresor-kek"        { capabilities = ["update"] }
path "transit/decrypt/tresor-kek"        { capabilities = ["update"] }
path "transit/hmac/tresor-kek/sha2-256"  { capabilities = ["update"] }
# ref+vault, within material.vault.allow
path "secret/data/duckdb/*"              { capabilities = ["read"] }
# client_auth: vault
path "transit/keys/zitadel-app"          { capabilities = ["read"] }
path "transit/sign/zitadel-app/sha2-256" { capabilities = ["update"] }

• Give hmac on the KEK to the service alone: whoever holds it computes the root.
• With state.auth: vault, read on database/creds/<role> too (below).
• Admins write ref+vault references; the service reads them. Keep read under the prefixes the allowlist names.

The database's credentials

The state's database (PostgreSQL, SQL Server) logs the service in with credentials a database secrets engine makes for it (spec 027): no password stored anywhere, a user of its own for each lease.

vault: {address: https://bao.example:8200, auth: {method: kubernetes, role: tresor}}
state:
  kind: postgres
  dsn: host=pg.example dbname=tresor sslmode=verify-full     # no user: the engine's is taken
  auth: vault
  vault: {mount: database, role: tresor, set_role: tresor_owner}   # source: <name> for a named vault source's connection

Each lease is another user. On PostgreSQL every one of them is granted one group role, which the service sets on each connection (set_role, required there): it owns the tables, so the next lease's user reads and writes them, and Vault drops an expired user cleanly. Once, as the database's owner:

CREATE ROLE tresor_owner NOLOGIN;
GRANT ALL ON DATABASE tresor TO tresor_owner;
GRANT ALL ON SCHEMA public TO tresor_owner;

and the engine's role:

bao write database/roles/tresor db_name=tresor-pg default_ttl=1h max_ttl=24h \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}' IN ROLE tresor_owner;"

On SQL Server the tables are the schema's: give each lease's user a database role (db_owner, or one with the rights the migrations need) in its creation statements; there is no set_role.

• How: GET <mount>/creds/<role> gives a user and a password with a lease. New connections take them while a connection made with them ends before the lease does (connections live 30 minutes); then the service takes new ones. Old leases are left to expire: a connection still open with them keeps working, then closes.
• The role's default_ttl must be more than 32 minutes (an hour or more is usual); a shorter lease is refused, with what to raise. Each replica takes credentials of its own, about once per default_ttl minus 32 minutes.
• The policy: path "database/creds/tresor" { capabilities = ["read"] }.
• Nothing of the credentials is logged: the path and the lease's length only.

The chart

vaultToken: {enabled: true, audience: vault}     # a projected token at /var/run/tresor/vault-token/token
extraVolumes: [{name: ca, configMap: {name: vault-ca}}]
extraVolumeMounts: [{name: ca, mountPath: /etc/tresor-ca, readOnly: true}]
config:
  state: {kind: kubernetes}
  keys: {kind: vault, key: tresor-kek}
  vault:
    address: https://bao.bao.svc:8200
    ca_file: /etc/tresor-ca/ca.crt               # Vault's private CA
    auth: {method: kubernetes, role: tresor}     # jwt_file set by the chart (vaultToken)
  material:
    vault: {allow: [{mount: secret, prefixes: [duckdb/]}]}

• vaultToken mounts a projected ServiceAccount token, bound to audience, and sets vault.auth.jwt_file to it. jwt needs it (or a jwt_file of your own).
• With kubernetes and no vaultToken, the pod's own ServiceAccount token is mounted. It is good for the Kubernetes API too: prefer vaultToken.
• ca_file is Vault's alone. When the issuer has a private CA too, add SSL_CERT_DIR in env.

scripts/ci/kind.sh checks this in CI: OpenBao in the cluster with Kubernetes auth and the projected token, the KEK in Transit, a ref+vault reference read through the protocol, on the Kubernetes store.

A second Vault

Another Vault (another region, another team's) is a named source: material.sources with kind: vault and a vault: block of its own. References name it: ref+vault-us://….

• With the chart's vaultToken, a named source's kubernetes or jwt login gets the same token file. Each Vault's role must accept its audience.
• The KEK and client_auth: vault use the top-level vault: only.

Failures

• Fail closed. A Vault that does not answer gives 503, never an empty value.
• A data key Transit says is not its own (a retired version, a ciphertext that does not open) is sealed.
• A 403 with a live token is the policy's refusal: it costs no new login.