HashiCorp Vault holds every credential the cluster uses, from the Cloudflare zone token to database passwords. It creates a bootstrap ordering problem: ArgoCD reads its own secrets from Vault via ESO, so Vault must exist before ArgoCD can manage anything. The resolution is documented directly in the argocd-apps values.
ArgoCD configuration
Excerpt from argocd-apps/values.yaml in the argocd-apps chart,
with annotations added for this site:
vault:
application: true
project: k3s-services
# false, and the template syncs with no prune so the adopted release is never deleted
autoSync: false
# Bootstrapped via GitLab CI before ArgoCD exists (ArgoCD reads its
# own secrets from Vault via ESO). ArgoCD adopts the existing release
# on first sync — no prune so it doesn't delete anything CI created.
Chart values
The full vault/values.yaml from the service's
own chart:
# ─────────────────────────────────────────────────────────────────────
# Vault deployment — opi5-cluster
#
# This wrapper installs the upstream `hashicorp/vault` chart as a Helm
# dependency. Helm passes values to a dependency ONLY under the
# dependency's key, so ALL upstream chart values live under `vault:`
# below. GitLab CI bootstraps Vault before ArgoCD exists (ArgoCD reads
# its own secrets from Vault via ESO), then ArgoCD adopts it — both
# paths render THIS wrapper, so they cannot drift.
#
# `bootstrap:` and the templates/ files are wrapper-only; everything the
# upstream chart reads is under `vault:`.
# ─────────────────────────────────────────────────────────────────────
vault:
global:
enabled: true
# The namespace to deploy to. Defaults to the `helm` installation namespace.
namespace: ""
# TLS for end-to-end encrypted transport
tlsDisable: true
# External vault server address for the injector and CSI provider to use.
externalVaultAddr: ""
openshift: false
# Create PodSecurityPolicy for pods
psp:
enable: false
serverTelemetry:
prometheusOperator: false
injector:
enabled: false
server:
enabled: true
enterpriseLicense:
secretName: ""
image:
repository: hashicorp/vault
tag: "2.0.4"
pullPolicy: IfNotPresent
updateStrategyType: "OnDelete"
logLevel: "info"
logFormat: "standard"
resources: {}
ingress:
enabled: false
annotations:
istio.ingress.kubernetes.io/router.entrypoints: websecure
ingressClassName: istio
pathType: Prefix
activeService: true
hosts:
- host: vault.opi5cluster.co.uk
paths:
- /
tls:
- secretName: opi5cluster-co-uk-domain-secret
hosts:
- vault.opi5cluster.co.uk
route:
enabled: false
authDelegator:
enabled: true
readinessProbe:
enabled: true
port: 8200
failureThreshold: 2
initialDelaySeconds: 5
periodSeconds: 5
successThreshold: 1
timeoutSeconds: 3
livenessProbe:
enabled: false
tolerations:
- key: node-role.kubernetes.io/master
operator: Exists
effect: NoSchedule
# Headless service used by the Vault Statefulset.
service:
enabled: true
active:
enabled: true
standby:
enabled: false
instanceSelector:
enabled: true
type: ClusterIP
port: 8200
targetPort: 8200
dataStorage:
enabled: true
size: 10Gi
mountPath: "/vault/data"
storageClass: longhorn-ssd-small
accessMode: ReadWriteOnce
persistentVolumeClaimRetentionPolicy:
whenDeleted: Retain
whenScaled: Retain
auditStorage:
enabled: false
dev:
enabled: false
standalone:
enabled: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "raft" {
path = "/vault/data"
}
service_registration "kubernetes" {}
# HA mode is intentionally disabled — Vault runs as a single-node
# standalone with integrated Raft storage. Enabling this requires a
# storage migration (see comments above), so leave it off.
ha:
enabled: false
replicas: 3
apiAddr: null
clusterAddr: null
raft:
enabled: true
setNodeId: true
config: |
ui = true
listener "tcp" {
tls_disable = 1
address = "[::]:8200"
cluster_address = "[::]:8201"
}
storage "raft" {
path = "/vault/data"
}
service_registration "kubernetes" {}
# Monitoring: expose Prometheus metrics at
# /v1/sys/metrics?format=prometheus (scraped by the `monitoring`
# Alloy DaemonSet → self-hosted Prometheus). unauthenticated_metrics_access
# allows scraping without a token on this internal-only listener;
# remove it and scrape with a token if the listener is ever exposed.
telemetry {
disable_hostname = true
prometheus_retention_time = "30s"
unauthenticated_metrics_access = true
}
# Auto-unseal sidecar: reads the shamir keys from the `unseal-keys`
# secret and unseals Vault on every pod start, so restarts and node
# reboots don't require manual key entry. Create once:
# kubectl -n vault create secret generic unseal-keys \
# --from-literal=key1='...' --from-literal=key2='...' --from-literal=key3='...'
extraContainers:
- name: vault-unsealer
image: hashicorp/vault:1.20.1
imagePullPolicy: IfNotPresent
command: ["/bin/sh", "-ec"]
args:
- |
export VAULT_ADDR=http://127.0.0.1:8200
# Poll Vault's seal status. No `|| true`: surface errors so the
# sidecar logs show exactly what's wrong instead of silently
# swallowing them.
while true; do
if vault status -format=json 2>&1 | grep -Eq '"sealed": ?true'; then
echo "vault sealed, unsealing"
vault operator unseal "$(cat /vault/unseal/key1)"
vault operator unseal "$(cat /vault/unseal/key2)"
vault operator unseal "$(cat /vault/unseal/key3)"
echo "unseal attempt complete"
fi
sleep 10
done
volumeMounts:
- name: userconfig-unseal-keys
mountPath: /vault/unseal
readOnly: true
# Chart helper renders secret `unseal-keys` as volume `userconfig-unseal-keys`.
extraVolumes:
- name: unseal-keys
type: secret
defaultMode: 420
serviceAccount:
create: true
name: "vault-service-account"
createSecret: true
serviceDiscovery:
enabled: true
ui:
enabled: true
publishNotReadyAddresses: true
activeVaultPodOnly: false
serviceType: "ClusterIP"
serviceNodePort: null
externalPort: 8200
targetPort: 8200
csi:
enabled: false
serverTelemetry:
serviceMonitor:
enabled: false
prometheusRules:
enabled: false
# Wrapper-only (not part of the upstream chart)
# Bootstrap Job configuration (templates/bootstrap-job.yaml).
bootstrap:
# GitLab group/project namespace_id for the jwt role (CI secrets).
Manifests & templates
templates/bootstrap-job.yaml
One-shot bootstrap Job seeding Vault with the cluster secrets on first install.
Show manifest
{{- /*
Bootstrap Job — turns the once-manual Vault setup (kv mount, policies,
userpass, kubernetes auth, jwt auth) into version-controlled, idempotent
Helm resources. Runs on every install/upgrade via Helm hooks; ArgoCD
honors these.
Requires a Secret named `vault-bootstrap` in the `vault` namespace with
keys `root-token` (an admin-capable Vault token) and `admin-password`
(the userpass password for admin_user). Create it once:
kubectl -n vault create secret generic vault-bootstrap \
--from-literal=root-token='...' \
--from-literal=admin-password='...'
*/}}
apiVersion: v1
kind: ServiceAccount
metadata:
name: vault-bootstrap
namespace: vault
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-weight": "-20"
# Persist: the long-lived token Secret references this SA.
"helm.sh/hook-delete-policy": before-hook-creation
---
# Long-lived token for vault-bootstrap, used as Vault's token_reviewer_jwt.
# The Job's own projected SA token expires shortly after the pod completes,
# which breaks Vault's TokenReview (→ auth/kubernetes/login "permission
# denied"). A static kubernetes.io/service-account-token Secret stays valid
# until deleted.
apiVersion: v1
kind: Secret
metadata:
name: vault-bootstrap-token
namespace: vault
annotations:
kubernetes.io/service-account.name: vault-bootstrap
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-weight": "0"
# Persist across syncs: the token must stay stable because Vault's
# token_reviewer_jwt references it. Deleting it after the hook (as with
# hook-succeeded) invalidates TokenReview on the next sync.
"helm.sh/hook-delete-policy": before-hook-creation
type: kubernetes.io/service-account-token
---
# Grants the bootstrap SA's token the ability to perform TokenReview,
# which Vault's kubernetes auth config requires as token_reviewer_jwt.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: vault-bootstrap-tokenreview
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-weight": "-10"
# Persist: the bootstrap SA needs TokenReview permission on re-runs.
"helm.sh/hook-delete-policy": before-hook-creation
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: system:auth-delegator
subjects:
- kind: ServiceAccount
name: vault-bootstrap
namespace: vault
---
apiVersion: batch/v1
kind: Job
metadata:
name: vault-bootstrap
namespace: vault
annotations:
"helm.sh/hook": post-install,post-upgrade
"helm.sh/hook-weight": "10"
"helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
spec:
template:
metadata:
name: vault-bootstrap
spec:
restartPolicy: Never
serviceAccountName: vault-bootstrap
containers:
- name: bootstrap
image: hashicorp/vault:1.20.1
command: ["/bin/sh", "-ec"]
args:
- |
export VAULT_ADDR=http://vault:8200
export VAULT_TOKEN="${VAULT_ROOT_TOKEN}"
# Wait until Vault is reachable and unsealed (60x2s).
for i in $(seq 1 60); do
if vault status -format=json 2>/dev/null | grep -q '"sealed":false'; then
break
fi
sleep 2
done
vault status >/dev/null 2>&1 || { echo "Vault not ready/unsealed after 120s"; exit 1; }
echo "== kv mount =="
vault secrets list -format=json 2>/dev/null | grep -q '"opi5-cluster/"' || vault secrets enable -path=opi5-cluster -version=2 kv
echo "== policies =="
vault policy read admin >/dev/null 2>&1 || printf '%s' $'path "*" {\n capabilities = ["create", "read", "update", "delete", "list", "sudo"]\n}\n' | vault policy write admin -
vault policy read reader >/dev/null 2>&1 || printf '%s' $'path "*" {\n capabilities = ["read", "list"]\n}\n' | vault policy write reader -
echo "== userpass =="
vault auth list -format=json 2>/dev/null | grep -q '"userpass/"' || vault auth enable userpass
vault write "auth/userpass/users/${ADMIN_USER}" password="${ADMIN_PASSWORD}" policies="admin"
echo "== kubernetes auth =="
vault auth list -format=json 2>/dev/null | grep -q '"kubernetes/"' || vault auth enable kubernetes
# Host/cert are self-served from the job's own service account.
# token_reviewer_jwt uses the long-lived vault-bootstrap-token
# (a projected SA token would expire and break TokenReview).
vault write auth/kubernetes/config \
token_reviewer_jwt="$(cat /var/run/secrets/bootstrap-token/token)" \
kubernetes_host="https://kubernetes.default.svc.cluster.local:443" \
kubernetes_ca_cert="$(cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt)" \
disable_issuer_verification=true
# audience is derived from the job's own SA token so it always
# matches the cluster's actual token audience (hardcoding it
# broke ESO login with "audience claim does not match").
# Decoded in pure shell (vault image has no python/jq/-E).
AUDIENCE="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token \
| cut -d. -f2 \
| tr '_-' '/+' \
| base64 -d 2>/dev/null \
| sed -n 's/.*"aud":\[*"\([^"]*\)".*/\1/p')"
vault write auth/kubernetes/role/k3s-role \
bound_service_account_names="*" \
bound_service_account_namespaces="*" \
policies=reader \
ttl=1h \
audience="${AUDIENCE}"
echo "== jwt (GitLab CI) =="
vault auth list -format=json 2>/dev/null | grep -q '"jwt/"' || vault auth enable jwt
vault write auth/jwt/config \
oidc_discovery_url="https://gitlab.com" \
bound_issuer="https://gitlab.com"
printf '%s' $'{\n "role_type": "jwt",\n "policies": ["reader"],\n "token_explicit_max_ttl": 60,\n "user_claim": "user_email",\n "bound_audiences": "https://vault.opi5cluster.co.uk",\n "bound_claims_type": "glob",\n "bound_claims": {\n "namespace_id": "{{ .Values.bootstrap.gitlabNamespaceId }}"\n }\n}\n' | vault write auth/jwt/role/gitlab-role -
echo "bootstrap complete"
env:
- name: VAULT_ROOT_TOKEN
valueFrom:
secretKeyRef:
name: vault-bootstrap
key: root-token
- name: ADMIN_PASSWORD
valueFrom:
secretKeyRef:
name: vault-bootstrap
key: admin-password
- name: ADMIN_USER
valueFrom:
secretKeyRef:
name: vault-bootstrap
key: admin-username
volumeMounts:
- name: bootstrap-token
mountPath: /var/run/secrets/bootstrap-token
readOnly: true
volumes:
- name: bootstrap-token
secret:
secretName: vault-bootstrap-token
templates/cluster-secret-store.yaml
ClusterSecretStore binding External Secrets Operator to Vault.
Show manifest
apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
name: vault-cluster-secret-store
spec:
provider:
vault:
server: http://vault.vault.svc.cluster.local:8200
path: opi5-cluster
version: v2
auth:
kubernetes:
mountPath: kubernetes
role: k3s-role
templates/http-route.yaml
Gateway API HTTPRoute exposing the service through the Istio gateway under opi5cluster.co.uk.
Show manifest
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: vault-httproute
namespace: vault
annotations:
link.argocd.argoproj.io/external-link: "https://vault.opi5cluster.co.uk"
spec:
parentRefs:
- name: istio-gateway
namespace: istio
sectionName: websecure
hostnames:
- vault.opi5cluster.co.uk
rules:
- backendRefs:
- name: vault-ui
port: 8200
templates/role-and-role-binding.yaml
RBAC for Vault's Kubernetes auth role.
Show manifest
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: vault
name: vault-role
rules:
- apiGroups: [""]
resources: ["pods"]
verbs: ["get", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: vault-role-binding
namespace: vault
labels:
app: vault
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: vault-role
subjects:
- kind: ServiceAccount
name: vault-service-account
namespace: vault
- ArgoCD adopts the CI-installed release on first sync with no prune, so Git never deletes what bootstrap created.
- The bootstrap Job is what makes the chicken-and-egg work: ArgoCD reads its own secrets from Vault via ESO, so Vault lands first, seeded by GitLab CI.
Trade-offs
Decision. Bootstrap Vault via GitLab CI, then let ArgoCD adopt the release
Alternative. Installing Vault manually outside GitOps forever
Why. Adoption gives Git-managed drift correction without a chicken-and-egg deadlock at the bottom of the dependency chain.
Decision. Sync with prune disabled for this one app
Alternative. Standard prune behaviour
Why. A prune against the bootstrapped release could delete resources CI created and break every secret in the cluster.
← Back to Platform & Infrastructure · All service groups