Vault

Keeps every credential the cluster uses in one audited place

HashiCorp Vault holds every credential the cluster uses, from the Cloudflare zone token to database passwords. It creates a bootstrap ordering problem: ArgoCD reads its own secrets from Vault via ESO, so Vault must exist before ArgoCD can manage anything. The resolution is documented directly in the argocd-apps values.

ArgoCD configuration

Excerpt from argocd-apps/values.yaml in the argocd-apps chart, with annotations added for this site:

vault:
  application: true
  project: k3s-services
  # false, and the template syncs with no prune so the adopted release is never deleted
  autoSync: false
  # Bootstrapped via GitLab CI before ArgoCD exists (ArgoCD reads its
  # own secrets from Vault via ESO). ArgoCD adopts the existing release
  # on first sync — no prune so it doesn't delete anything CI created.

Chart values

The full vault/values.yaml from the service's own chart:

# ─────────────────────────────────────────────────────────────────────
# Vault deployment — opi5-cluster
#
# This wrapper installs the upstream `hashicorp/vault` chart as a Helm
# dependency. Helm passes values to a dependency ONLY under the
# dependency's key, so ALL upstream chart values live under `vault:`
# below. GitLab CI bootstraps Vault before ArgoCD exists (ArgoCD reads
# its own secrets from Vault via ESO), then ArgoCD adopts it — both
# paths render THIS wrapper, so they cannot drift.
#
# `bootstrap:` and the templates/ files are wrapper-only; everything the
# upstream chart reads is under `vault:`.
# ─────────────────────────────────────────────────────────────────────

vault:
  global:
    enabled: true

    # The namespace to deploy to. Defaults to the `helm` installation namespace.
    namespace: ""

    # TLS for end-to-end encrypted transport
    tlsDisable: true

    # External vault server address for the injector and CSI provider to use.
    externalVaultAddr: ""

    openshift: false

    # Create PodSecurityPolicy for pods
    psp:
      enable: false

    serverTelemetry:
      prometheusOperator: false

  injector:
    enabled: false

  server:
    enabled: true
    enterpriseLicense:
      secretName: ""

    image:
      repository: hashicorp/vault
      tag: "2.0.4"
      pullPolicy: IfNotPresent
    updateStrategyType: "OnDelete"

    logLevel: "info"
    logFormat: "standard"

    resources: {}

    ingress:
      enabled: false
      annotations:
        istio.ingress.kubernetes.io/router.entrypoints: websecure
      ingressClassName: istio
      pathType: Prefix
      activeService: true
      hosts:
        - host: vault.opi5cluster.co.uk
          paths:
            - /
      tls:
        - secretName: opi5cluster-co-uk-domain-secret
          hosts:
            - vault.opi5cluster.co.uk

    route:
      enabled: false

    authDelegator:
      enabled: true

    readinessProbe:
      enabled: true
      port: 8200
      failureThreshold: 2
      initialDelaySeconds: 5
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 3

    livenessProbe:
      enabled: false

    tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule


    # Headless service used by the Vault Statefulset.
    service:
      enabled: true
      active:
        enabled: true
      standby:
        enabled: false
      instanceSelector:
        enabled: true
      type: ClusterIP
      port: 8200
      targetPort: 8200

    dataStorage:
      enabled: true
      size: 10Gi
      mountPath: "/vault/data"
      storageClass: longhorn-ssd-small
      accessMode: ReadWriteOnce

    persistentVolumeClaimRetentionPolicy:
      whenDeleted: Retain
      whenScaled: Retain

    auditStorage:
      enabled: false

    dev:
      enabled: false

    standalone:
      enabled: true

      config: |
        ui = true

        listener "tcp" {
          tls_disable = 1
          address = "[::]:8200"
          cluster_address = "[::]:8201"
        }

        storage "raft" {
          path = "/vault/data"
        }

        service_registration "kubernetes" {}

    # HA mode is intentionally disabled — Vault runs as a single-node
    # standalone with integrated Raft storage. Enabling this requires a
    # storage migration (see comments above), so leave it off.
    ha:
      enabled: false
      replicas: 3
      apiAddr: null
      clusterAddr: null

      raft:
        enabled: true
        setNodeId: true

        config: |
          ui = true

          listener "tcp" {
            tls_disable = 1
            address = "[::]:8200"
            cluster_address = "[::]:8201"
          }

          storage "raft" {
            path = "/vault/data"
          }

          service_registration "kubernetes" {}

          # Monitoring: expose Prometheus metrics at
          # /v1/sys/metrics?format=prometheus (scraped by the `monitoring`
          # Alloy DaemonSet → self-hosted Prometheus). unauthenticated_metrics_access
          # allows scraping without a token on this internal-only listener;
          # remove it and scrape with a token if the listener is ever exposed.
          telemetry {
            disable_hostname = true
            prometheus_retention_time = "30s"
            unauthenticated_metrics_access = true
          }

    # Auto-unseal sidecar: reads the shamir keys from the `unseal-keys`
    # secret and unseals Vault on every pod start, so restarts and node
    # reboots don't require manual key entry. Create once:
    #   kubectl -n vault create secret generic unseal-keys \
    #     --from-literal=key1='...' --from-literal=key2='...' --from-literal=key3='...'
    extraContainers:
      - name: vault-unsealer
        image: hashicorp/vault:1.20.1
        imagePullPolicy: IfNotPresent
        command: ["/bin/sh", "-ec"]
        args:
          - |
            export VAULT_ADDR=http://127.0.0.1:8200
            # Poll Vault's seal status. No `|| true`: surface errors so the
            # sidecar logs show exactly what's wrong instead of silently
            # swallowing them.
            while true; do
              if vault status -format=json 2>&1 | grep -Eq '"sealed": ?true'; then
                echo "vault sealed, unsealing"
                vault operator unseal "$(cat /vault/unseal/key1)"
                vault operator unseal "$(cat /vault/unseal/key2)"
                vault operator unseal "$(cat /vault/unseal/key3)"
                echo "unseal attempt complete"
              fi
              sleep 10
            done
        volumeMounts:
          - name: userconfig-unseal-keys
            mountPath: /vault/unseal
            readOnly: true
    # Chart helper renders secret `unseal-keys` as volume `userconfig-unseal-keys`.
    extraVolumes:
      - name: unseal-keys
        type: secret
        defaultMode: 420

    serviceAccount:
      create: true
      name: "vault-service-account"
      createSecret: true
      serviceDiscovery:
        enabled: true

  ui:
    enabled: true
    publishNotReadyAddresses: true
    activeVaultPodOnly: false
    serviceType: "ClusterIP"
    serviceNodePort: null
    externalPort: 8200
    targetPort: 8200

  csi:
    enabled: false

  serverTelemetry:
    serviceMonitor:
      enabled: false
    prometheusRules:
      enabled: false

# Wrapper-only (not part of the upstream chart)

# Bootstrap Job configuration (templates/bootstrap-job.yaml).
bootstrap:
  # GitLab group/project namespace_id for the jwt role (CI secrets).

Manifests & templates

templates/bootstrap-job.yaml

One-shot bootstrap Job seeding Vault with the cluster secrets on first install.

Show manifest
{{- /*
Bootstrap Job — turns the once-manual Vault setup (kv mount, policies,
userpass, kubernetes auth, jwt auth) into version-controlled, idempotent
Helm resources. Runs on every install/upgrade via Helm hooks; ArgoCD
honors these.

Requires a Secret named `vault-bootstrap` in the `vault` namespace with
keys `root-token` (an admin-capable Vault token) and `admin-password`
(the userpass password for admin_user). Create it once:
  kubectl -n vault create secret generic vault-bootstrap \
    --from-literal=root-token='...' \
    --from-literal=admin-password='...'
*/}}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: vault-bootstrap
  namespace: vault
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "-20"
    # Persist: the long-lived token Secret references this SA.
    "helm.sh/hook-delete-policy": before-hook-creation
---
# Long-lived token for vault-bootstrap, used as Vault's token_reviewer_jwt.
# The Job's own projected SA token expires shortly after the pod completes,
# which breaks Vault's TokenReview (→ auth/kubernetes/login "permission
# denied"). A static kubernetes.io/service-account-token Secret stays valid
# until deleted.
apiVersion: v1
kind: Secret
metadata:
  name: vault-bootstrap-token
  namespace: vault
  annotations:
    kubernetes.io/service-account.name: vault-bootstrap
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "0"
    # Persist across syncs: the token must stay stable because Vault's
    # token_reviewer_jwt references it. Deleting it after the hook (as with
    # hook-succeeded) invalidates TokenReview on the next sync.
    "helm.sh/hook-delete-policy": before-hook-creation
type: kubernetes.io/service-account-token
---
# Grants the bootstrap SA's token the ability to perform TokenReview,
# which Vault's kubernetes auth config requires as token_reviewer_jwt.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: vault-bootstrap-tokenreview
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "-10"
    # Persist: the bootstrap SA needs TokenReview permission on re-runs.
    "helm.sh/hook-delete-policy": before-hook-creation
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: system:auth-delegator
subjects:
  - kind: ServiceAccount
    name: vault-bootstrap
    namespace: vault
---
apiVersion: batch/v1
kind: Job
metadata:
  name: vault-bootstrap
  namespace: vault
  annotations:
    "helm.sh/hook": post-install,post-upgrade
    "helm.sh/hook-weight": "10"
    "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
  template:
    metadata:
      name: vault-bootstrap
    spec:
      restartPolicy: Never
      serviceAccountName: vault-bootstrap
      containers:
        - name: bootstrap
          image: hashicorp/vault:1.20.1
          command: ["/bin/sh", "-ec"]
          args:
            - |
              export VAULT_ADDR=http://vault:8200
              export VAULT_TOKEN="${VAULT_ROOT_TOKEN}"

              # Wait until Vault is reachable and unsealed (60x2s).
              for i in $(seq 1 60); do
                if vault status -format=json 2>/dev/null | grep -q '"sealed":false'; then
                  break
                fi
                sleep 2
              done
              vault status >/dev/null 2>&1 || { echo "Vault not ready/unsealed after 120s"; exit 1; }

              echo "== kv mount =="
              vault secrets list -format=json 2>/dev/null | grep -q '"opi5-cluster/"' || vault secrets enable -path=opi5-cluster -version=2 kv

              echo "== policies =="
              vault policy read admin >/dev/null 2>&1 || printf '%s' $'path "*" {\n  capabilities = ["create", "read", "update", "delete", "list", "sudo"]\n}\n' | vault policy write admin -
              vault policy read reader >/dev/null 2>&1 || printf '%s' $'path "*" {\n  capabilities = ["read", "list"]\n}\n' | vault policy write reader -

              echo "== userpass =="
              vault auth list -format=json 2>/dev/null | grep -q '"userpass/"' || vault auth enable userpass
              vault write "auth/userpass/users/${ADMIN_USER}" password="${ADMIN_PASSWORD}" policies="admin"

              echo "== kubernetes auth =="
              vault auth list -format=json 2>/dev/null | grep -q '"kubernetes/"' || vault auth enable kubernetes
              # Host/cert are self-served from the job's own service account.
              # token_reviewer_jwt uses the long-lived vault-bootstrap-token
              # (a projected SA token would expire and break TokenReview).
              vault write auth/kubernetes/config \
                token_reviewer_jwt="$(cat /var/run/secrets/bootstrap-token/token)" \
                kubernetes_host="https://kubernetes.default.svc.cluster.local:443" \
                kubernetes_ca_cert="$(cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt)" \
                disable_issuer_verification=true

              # audience is derived from the job's own SA token so it always
              # matches the cluster's actual token audience (hardcoding it
              # broke ESO login with "audience claim does not match").
              # Decoded in pure shell (vault image has no python/jq/-E).
              AUDIENCE="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token \
                | cut -d. -f2 \
                | tr '_-' '/+' \
                | base64 -d 2>/dev/null \
                | sed -n 's/.*"aud":\[*"\([^"]*\)".*/\1/p')"
              vault write auth/kubernetes/role/k3s-role \
                bound_service_account_names="*" \
                bound_service_account_namespaces="*" \
                policies=reader \
                ttl=1h \
                audience="${AUDIENCE}"

              echo "== jwt (GitLab CI) =="
              vault auth list -format=json 2>/dev/null | grep -q '"jwt/"' || vault auth enable jwt
              vault write auth/jwt/config \
                oidc_discovery_url="https://gitlab.com" \
                bound_issuer="https://gitlab.com"
              printf '%s' $'{\n  "role_type": "jwt",\n  "policies": ["reader"],\n  "token_explicit_max_ttl": 60,\n  "user_claim": "user_email",\n  "bound_audiences": "https://vault.opi5cluster.co.uk",\n  "bound_claims_type": "glob",\n  "bound_claims": {\n    "namespace_id": "{{ .Values.bootstrap.gitlabNamespaceId }}"\n  }\n}\n' | vault write auth/jwt/role/gitlab-role -

              echo "bootstrap complete"
          env:
            - name: VAULT_ROOT_TOKEN
              valueFrom:
                secretKeyRef:
                  name: vault-bootstrap
                  key: root-token
            - name: ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: vault-bootstrap
                  key: admin-password
            - name: ADMIN_USER
              valueFrom:
                secretKeyRef:
                  name: vault-bootstrap
                  key: admin-username
          volumeMounts:
            - name: bootstrap-token
              mountPath: /var/run/secrets/bootstrap-token
              readOnly: true
      volumes:
        - name: bootstrap-token
          secret:
            secretName: vault-bootstrap-token

templates/cluster-secret-store.yaml

ClusterSecretStore binding External Secrets Operator to Vault.

Show manifest
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: vault-cluster-secret-store
spec:
  provider:
    vault:
      server: http://vault.vault.svc.cluster.local:8200
      path: opi5-cluster
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes
          role: k3s-role

templates/http-route.yaml

Gateway API HTTPRoute exposing the service through the Istio gateway under opi5cluster.co.uk.

Show manifest
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: vault-httproute
  namespace: vault
  annotations:
    link.argocd.argoproj.io/external-link: "https://vault.opi5cluster.co.uk"
spec:
  parentRefs:
    - name: istio-gateway
      namespace: istio
      sectionName: websecure
  hostnames:
    - vault.opi5cluster.co.uk
  rules:
    - backendRefs:
        - name: vault-ui
          port: 8200

templates/role-and-role-binding.yaml

RBAC for Vault's Kubernetes auth role.

Show manifest
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: vault
  name: vault-role
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: vault-role-binding
  namespace: vault
  labels:
    app: vault
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: vault-role
subjects:
- kind: ServiceAccount
  name: vault-service-account
  namespace: vault
  • ArgoCD adopts the CI-installed release on first sync with no prune, so Git never deletes what bootstrap created.
  • The bootstrap Job is what makes the chicken-and-egg work: ArgoCD reads its own secrets from Vault via ESO, so Vault lands first, seeded by GitLab CI.

Trade-offs

Decision. Bootstrap Vault via GitLab CI, then let ArgoCD adopt the release

Alternative. Installing Vault manually outside GitOps forever

Why. Adoption gives Git-managed drift correction without a chicken-and-egg deadlock at the bottom of the dependency chain.

Decision. Sync with prune disabled for this one app

Alternative. Standard prune behaviour

Why. A prune against the bootstrapped release could delete resources CI created and break every secret in the cluster.

← Back to Platform & Infrastructure · All service groups