Istio

Istio replaced Traefik as the cluster edge on 2026-09-20. A shared Gateway (istio-gateway on MetalLB 192.168.1.10) terminates TLS for every opi5cluster.co.uk host and forwards raw TCP for postgres, kafka, and torrent traffic. On top of that, the ambient mesh (ztunnel node proxies, no sidecars) gives every enrolled workload automatic mTLS with SPIFFE identity. Kiali renders the mesh graph; the whole thing is one values-only wrapper chart over the five pinned upstream Istio charts.

ArgoCD configuration

Excerpt from argocd-apps/values.yaml in the argocd-apps chart, with annotations added for this site:

istio:
  # true: standard app-of-apps entry (replaced the traefik entry 2026-09-20)
  application: true
  project: k3s-services
  # false: mesh + gateway changes are deliberate, review-first events
  autoSync: false

Chart values

The full istio/values.yaml from the service's own chart:

# =============================================================================
# istio wrapper chart — single source of Istio config for opi5-cluster.
# ArgoCD ApplicationSet deploys this chart with release name "istio" into
# namespace "istio" (NOT istio-system — deliberate deviation, spec constraint).
#
# Subchart values are TOP-LEVEL keys named after each dependency/alias
# (base, istiod, cni, ztunnel, istio-gateway). The vendored 1.30.x upstream
# charts keep their builtin defaults under an internal `_internal_defaults_do_not_set`
# key and merge user values on top at render time (templates/zzz_profile.yaml),
# so anything set here overrides upstream defaults. Do NOT try to set values
# under `_internal_defaults_do_not_set` — that key is stripped before render.
# =============================================================================

# -----------------------------------------------------------------------------
# Global (shared across base/istiod/cni/ztunnel/istio-gateway)
# -----------------------------------------------------------------------------
global:
  # House convention: Istio lands in namespace `istio` (ApplicationSet
  # destination = app name), not istio-system.
  istioNamespace: istio
  # Official multi-arch (amd64/arm64) images — mandatory for the 4 arm64 Pi nodes.
  # Pinned, never floating :latest (house rule).
  hub: docker.io/istio
  tag: "1.30.4"

# -----------------------------------------------------------------------------
# base — CRDs + cluster-scoped RBAC only. No tunable values needed.
# NOTE: helm does NOT upgrade vendored dependency CRDs on `helm upgrade`.
# Minor-version upgrades need the manual CRD refresh documented in AGENTS.md.
# -----------------------------------------------------------------------------
base: {}

# -----------------------------------------------------------------------------
# istiod — control plane. Pinned to amd64 (x86) nodes per ARM budget.
# -----------------------------------------------------------------------------
istiod:
  # Applies files/profile-ambient.yaml: HBONE proxyMetadata, service scope
  # configs, distroless variant, ambient-aware defaults.
  profile: ambient
  # Explicit per-chart pin (ztunnel's internal default hub is registry.istio.io,
  # so per-chart pins guarantee every image comes from docker.io/istio @1.30.4).
  hub: docker.io/istio
  tag: "1.30.4"
  # Homelab: single replica, no HPA.
  autoscaleEnabled: false
  replicaCount: 2
  resources:
    requests:
      cpu: 250m
      memory: 1Gi
  # istiod is the only control-plane component that may avoid arm64 nodes.
  nodeSelector:
    kubernetes.io/arch: amd64
  env:
    # The gateway chart owns the Gateway's Deployment+Service (named
    # istio-gateway-istio; its pods get their proxyv2 image via the injection
    # webhook, which rewrites the chart's literal `image: auto`). Without
    # this, istiod's Gateway-API deploy controller ALSO provisions its own
    # <gateway>-<namespace> fleet that fights the chart service for MetalLB
    # IPs. The chart drops its gateway-provisioning RBAC rules when this is
    # "false".
    PILOT_ENABLE_GATEWAY_API_DEPLOYMENT_CONTROLLER: "false"
    # Alpha-tier Gateway API in Istio 1.30: required for the Gateway's TCP
    # listeners (postgres :5432, kafka :9092) backed by TCPRoute. istiod then
    # hard-watches TCPRoute/UDPRoute v1alpha2, which the cluster's Gateway API
    # CRDs only serve once they are the gateway-api v1.6.1 EXPERIMENTAL bundle
    # (swapped 2026-09-20, manual kubectl apply per house exception, documented
    # in ../traefik/). On the previous STANDARD bundle (v1alpha2 served:false)
    # this flag made the informer list fail ("could not find the requested
    # resource"), hung istiod readiness at 503 and wedged the rollout —
    # verified live 2026-09-19, reverted same day; unblocked 2026-09-20 by the
    # CRD swap. If the CRDs ever regress to the standard bundle, this flag
    # MUST go back to "false" or istiod will not start.
    PILOT_ENABLE_ALPHA_GATEWAY_API: "true"
  meshConfig:
    accessLogFile: /dev/stdout

# -----------------------------------------------------------------------------
# cni (istio-cni) — chained CNI plugin + ambient redirection. MUST run on all
# nodes (arm64 + amd64): no nodeSelector, no affinity overrides here.
# -----------------------------------------------------------------------------
cni:
  # Applies files/profile-ambient.yaml -> cni.ambient.enabled=true.
  profile: ambient
  hub: docker.io/istio
  tag: "1.30.4"
  # K3s with custom data-dir /mnt/ssd-small/rancher/k3s — verified live on
  # opi5-worker-0 (2026-09-18), NOT the upstream /opt/cni/bin + /etc/cni/net.d
  # defaults:
  #   - net.d dir contains 10-flannel.conflist (flannel is embedded in the k3s
  #     agent binary — there is no flannel DaemonSet to introspect via kubectl)
  #   - data/current/bin is the k3s-managed symlink to the live release's
  #     CNI plugin dir (flannel/portmap/loopback/bridge/host-local verified)
  # kubelet's --cni-bin-dir on this k3s install (verified live: sandbox
  # creation failed with "failed to find plugin istio-cni in path
  # [/mnt/ssd-small/rancher/k3s/data/cni]"). NOTE: this is NOT k3s's runtime
  # binary dir (data/current/bin) — that holds flannel/bridge/portmap, but
  # kubelet looks for CNI plugins in data/cni.
  cniBinDir: /mnt/ssd-small/rancher/k3s/data/cni
  cniConfDir: /mnt/ssd-small/rancher/k3s/agent/etc/cni/net.d
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 4
  # ARM budget: keep ambient overhead small on the 8 GB Pi nodes.
  resources:
    requests:
      cpu: 100m
      memory: 100Mi

# -----------------------------------------------------------------------------
# ztunnel — node proxy (L4 mTLS). DaemonSet on ALL nodes (arm64 + amd64):
# no nodeSelector, no affinity overrides here.
# -----------------------------------------------------------------------------
ztunnel:
  hub: docker.io/istio
  tag: "1.30.4"
  istioNamespace: istio
  resources:
    requests:
      cpu: 100m
      memory: 128Mi

istio-gateway:
  name: istio-gateway-istio
  replicaCount: 2
  autoscaling:
    enabled: false
  nodeSelector:
    kubernetes.io/arch: amd64
  service:
    type: LoadBalancer
    annotations:
      metallb.universe.tf/loadBalancerIPs: "192.168.1.10"
    ports:
      - name: http
        port: 80
        targetPort: 80
        protocol: TCP
        appProtocol: http
      - name: https
        port: 443
        targetPort: 443
        protocol: TCP
        appProtocol: https
      - name: tcp-postgres
        port: 5432
        targetPort: 5432
        protocol: TCP
      - name: tcp-kafka
        port: 9092
        targetPort: 9092
        protocol: TCP
      - name: tcp-torrent
        port: 6881
        targetPort: 6881
        protocol: TCP
      - name: http-envoy-prom
        port: 15090
        targetPort: 15090
        protocol: TCP
        appProtocol: http
  resources:
    requests:
      cpu: 100m
      memory: 128Mi

kiali:
  istio_namespace: istio
  auth:
    strategy: anonymous
  external_services:
    prometheus:
      url: http://monitoring-prometheus-server.monitoring.svc.cluster.local:9090
    grafana:
      # internal_url: what the kiali SERVER probes (health/version checks →
      # the "Grafana: Reachable" status and Add-ons health). external_url: what
      # the BROWSER opens (dashboard links) — the TLS route via istio-gateway
      # websecure-uk (monitoring repo templates/grafana-httproute.yaml).
      # Cluster-local DNS (.svc.cluster.local) is unresolvable from laptops,
      # so external_url must stay the public hostname. (Kiali 2.x renamed
      # in_cluster_url/url → internal_url/external_url; use the new keys.)
      # Anonymous VIEWER auth is enabled in the monitoring repo grafana config
      # (LAN-only cluster, house-approved Kiali integration path) — no
      # credentials in git, none needed here.
      enabled: true
      internal_url: http://monitoring-grafana.monitoring.svc.cluster.local:80
      external_url: https://grafana.opi5cluster.co.uk
      # Names MUST match the Grafana dashboard titles provisioned by the
      # monitoring repo (files/*.json); variables map kiali context to grafana
      # URL vars (var-*) — only list vars the dashboard actually declares.
      dashboards:
        - name: "Istio Mesh Dashboard"
        - name: "Istio Control Plane Dashboard"
        - name: "Istio Service Dashboard"
          variables:
            service: var-service
        - name: "Istio Workload Dashboard"
          variables:
            namespace: var-namespace
            workload: var-workload
        - name: "Istio Performance Dashboard"
        - name: "Istio Ztunnel Dashboard"
  # Graph default view: hide the "unknown" aggregate nodes. In ambient, ztunnel
  # reports L4 egress/probes by destination IP only (no SNI matching), so live
  # off-cluster traffic and kubelet/apiserver probes can never be attached to a
  # named service — they always aggregate under two "unknown" graph nodes
  # (Kiali FAQ: "Why are there many unknown nodes in the graph?"). Every real
  # destination is accounted for: named external hosts are registered as
  # ServiceEntries (templates/serviceentries.yaml) and the full inventory lives
  # in AGENTS.md. This preset only keeps the default graph clean; the nodes
  # come back with one click via the Hide toolbar.
  kiali_feature_flags:
    ui_defaults:
      graph:
        hide_options:
          - auto_select: true
            description: "Hide: unknown aggregate nodes (egress sink + probe source; see AGENTS.md inventory)"
            expression: "name = unknown"
  server:
    port: 20001
    web_root: /
  deployment:
    # discovery_selectors: hide unmeshed infra namespaces (longhorn's
    # runtime-created engine-image/instance-manager pods can't carry telemetry
    # labels -> "unknown" graph nodes; velero kopia-maintain Jobs are batch
    # noise). Kiali 2.x dropped api.namespaces.exclude — this is the v2 key.
    # GOTCHA: once ANY selector is set, kiali's built-in system-ns filter
    # (^(kube-.*|openshift.*|ibm.*|kiali-operator|istio-operator)) is
    # bypassed, so the kube-* namespaces must be excluded explicitly.
    discovery_selectors:
      default:
        - matchExpressions:
            - key: kubernetes.io/metadata.name
              operator: NotIn
              values: [kube-system, kube-node-lease, kube-public, velero, longhorn]
    pod_annotations:
      sidecar.istio.io/inject: "false"
    resources:
      requests:
        cpu: 100m
        memory: 128Mi

Manifests & templates

templates/gateway.yaml

The shared Gateway API Gateway: web :80, wildcard TLS websecure-uk and rustfs-uk :443, and raw TCP listeners for postgres :5432, kafka :9092, and torrent :6881.

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  # Name is load-bearing: app-repo HTTPRoutes parentRef name=istio-gateway
  # namespace=istio. The Service/Deployment are istio-gateway-istio — istiod
  # derives <gateway-name>-<gatewayClassName> and DNS-resolves it for status
  # (see values.yaml istio-gateway.name); monitoring scrapes that Service.
  name: istio-gateway
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
spec:
  gatewayClassName: istio
  listeners:
    # Plain HTTP edge (http->https redirects happen per-HTTPRoute, not here).
    - name: web
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All

    # Wildcard TLS termination, mirroring the old Traefik listeners.
    # Certs are cert-manager-issued (LE DNS01) and Reflector-replicated INTO
    # ns istio by the cert-manager repo; they must exist before listeners
    # can reach Programmed.
    - name: websecure-uk
      port: 443
      protocol: HTTPS
      hostname: "*.opi5cluster.co.uk"
      tls:
        mode: Terminate
        certificateRefs:
          - group: ""
            kind: Secret
            name: opi5cluster-co-uk-domain-secret
      allowedRoutes:
        namespaces:
          from: All

    - name: rustfs-uk
      port: 443
      protocol: HTTPS
      hostname: "*.rustfs.opi5cluster.co.uk"
      tls:
        mode: Terminate
        certificateRefs:
          - group: ""
            kind: Secret
            name: rustfs-subdomain-uk-secret
      allowedRoutes:
        namespaces:
          from: All

    # Raw TCP pass-through. Routes are Phase 2 (postgres pooler-rw 5432,
    # automq 9092) — listeners are live from Phase 1 with no routes attached.
    - name: postgres
      port: 5432
      protocol: TCP
      allowedRoutes:
        namespaces:
          from: All

    - name: kafka
      port: 9092
      protocol: TCP
      allowedRoutes:
        namespaces:
          from: All

    # Torrent (BitTorrent peer traffic). User decision 2026-09-20: TCP-ONLY —
    # no UDP listener (the Istio Gateway/TCPRoute has no UDP support) and no
    # dedicated LoadBalancer; the old qbittorrent LB on 192.168.1.6 is retired.
    # Route: qbittorrent-tcproute in the media-systems repo.
    - name: torrent
      port: 6881
      protocol: TCP
      allowedRoutes:
        namespaces:
          from: All

templates/serviceentries.yaml

External-destination registry: every off-cluster host the meshed workloads talk to (Discord, sports feeds, Dropbox, gitlab, the backup RustFS...), registered so Kiali telemetry attributes egress correctly.

{{- /*
External-destination registry (Group C of the 2026-09-20 full-service
inventory). Each entry names off-cluster destinations that meshed workloads
actually talk to, so they appear in Kiali's service inventory with the lock
icon instead of being unaccounted gray boxes.

TELEMETRY (verified 2026-09-21, correcting the older note here): istiod
resolves each SE hostname and pushes the resolved-IP→service map to
ztunnel, so ztunnel access logs + Kiali DO attribute matching egress to
the SE (e.g. `dst.service="discord.com" dst.workload="external-discord"`).
Residual traffic in the graph's "unknown" sink is (a) CDN edge-IP
rotation that istiod has not cached under the SE host — for Discord this
is now covered by enumerated SE `addresses` (see below); for other SEs, no
config fix; and (b) service-less pod-IP traffic (headless peers, direct
HBONE).
CLOSE-OUT (2026-09-21, cluster-wide socket sweep + ztunnel live logs): the
"new" unknown hub reported this day was NOT a new destination — every
currently-established external TCP socket in the cluster (bots bezos/
meridian/quote-my-shizzle/beacon-bot/whistle-bot/bouncer/wos-assistant-bot
included) rides a 162.159.13x.x Discord edge already enumerated below, and
fresh connections label correctly (dst.workload="external-discord"). The
residual graph edges are STALE LABELS: ztunnel fixes an attribution at
connection-open, so long-lived flows opened BEFORE the SE/address fix keep
their old `unknown` labels until they recycle (Discord gateway WS lives
hours-days). Liveness check: rate(istio_tcp_sent_bytes_total{
destination_service_name="unknown"}[5m]) — if a source shows there, exec
into the source pod and read /proc/net/tcp (established, non-10.x) to see
the real dst IPs before registering anything.
The entries buy: named telemetry, a queryable registry (Kiali services
list, istiod /debug/config_dump) and a policy anchor point if a
destination ever needs AuthorizationPolicy / egress control.

No trafficPolicy is set on purpose: these are pure registry names — egress
stays plaintext passthrough exactly as before.
*/ -}}

# discord.com / gateway.discord.gg / cdn.discordapp.com (Cloudflare-fronted).
# Talkers: meridian (Discord bot gateway), wos-assistant-bot, quote-my-shizzle,
# whistle-bot, beacon-bot, bezos, bouncer (socket-verified 2026-09-21: every
# bot's persistent gateway WS terminates on an edge IP enumerated below).
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-discord
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
    app.kubernetes.io/component: external-registry
  annotations:
    opi5cluster.co.uk/talkers: "meridian, wos-assistant-bot, quote-my-shizzle, whistle-bot, beacon-bot, bezos, bouncer"
spec:
  hosts:
    - discord.com
    - gateway.discord.gg
    - cdn.discordapp.com
  # Discord rides Cloudflare anycast (Spectrum) edges rotating inside
  # 162.159.128.0/20; host-resolution alone misses fresh edges (Kiali
  # unknown). GOTCHA: istiod's webhook rejects CIDRs on a DNS-resolution
  # SE ("CIDR addresses are allowed only for NONE/STATIC resolution
  # types"), so observed edge IPs are enumerated instead. Refresh by
  # re-digging the three hosts (or a ztunnel dst.addr sweep) when new
  # edges appear.
  addresses:
    - 162.159.128.233
    - 162.159.129.233
    - 162.159.130.233
    - 162.159.130.234
    - 162.159.133.233
    - 162.159.133.234
    - 162.159.134.233
    - 162.159.134.234
    - 162.159.135.232
    - 162.159.135.233
    - 162.159.135.234
    - 162.159.136.232
    - 162.159.136.234
    - 162.159.137.232
    - 162.159.138.232
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
# Sports data feeds pulled by the whistle producers (espn/f1 sit on Akamai,
# observed as 2.19.252.x egress).
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-whistle-feeds
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
    app.kubernetes.io/component: external-registry
  annotations:
    opi5cluster.co.uk/talkers: "whistle producers (all leagues)"
spec:
  hosts:
    - site.api.espn.com
    - sports.core.api.espn.com
    - a.espncdn.com
    - secure.espncdn.com
    - www.formula1.com
    - livetiming.formula1.com
    - api.the-odds-api.com
    - api.balldontlie.io
    - www.ufc.com
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
# Century Games WoS gift-code API + avatar CDN, polled by wos-assistant.
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-wos-game
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
    app.kubernetes.io/component: external-registry
  annotations:
    opi5cluster.co.uk/talkers: "wos-assistant (bot + gift ops)"
spec:
  hosts:
    - wos-giftcode.centurygame.com
    - wos-giftcode-api.centurygame.com
    - gof-formal-avatar.akamaized.net
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
# Dropbox image ingestion used by beacon-api (routes_images.py).
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-beacon-dropbox
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
    app.kubernetes.io/component: external-registry
  annotations:
    opi5cluster.co.uk/talkers: "beacon-api"
spec:
  hosts:
    - api.dropboxapi.com
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
# api-ninjas.com content APIs (facts/quotes): bouncer content service, trove.
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-api-ninjas
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
    app.kubernetes.io/component: external-registry
  annotations:
    opi5cluster.co.uk/talkers: "bouncer, trove"
spec:
  hosts:
    - api.api-ninjas.com
    - api-ninjas.com
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
# gitlab.com — gitlab-runner coordinator + CI job checkout/push traffic.
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-gitlab
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
    app.kubernetes.io/component: external-registry
  annotations:
    opi5cluster.co.uk/talkers: "gitlab-runner (+ ephemeral CI pods)"
spec:
  hosts:
    - gitlab.com
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS
---
# Standalone RustFS on backup-raspi3 (192.168.1.220, LAN) — S3-compatible
# backup endpoint for the CNPG Barman Cloud sidecars (WAL archiving + base
# backups to s3://postgres-backups/). Plain HTTP :9000 by design (accepted
# posture per postgres repo). Pod-IP egress here was the last unaccounted
# external destination in the Kiali graph (found via ztunnel unlabeled
# dst.addr sweep, 2026-09-21).
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-backup-rustfs
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
    app.kubernetes.io/component: external-registry
  annotations:
    opi5cluster.co.uk/talkers: "CNPG barman-cloud sidecars (ns postgres)"
spec:
  hosts:
    - backup-raspi3.opi5cluster.co.uk
  ports:
    - number: 9000
      name: http
      protocol: HTTP
  location: MESH_EXTERNAL
  resolution: DNS
---
# Optional search/answer providers configured for the whatsapp-bot assistant
# (tools.py; active only when the corresponding key is set).
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-whatsapp-search
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
    app.kubernetes.io/component: external-registry
  annotations:
    opi5cluster.co.uk/talkers: "whatsapp-bot (conditional providers)"
spec:
  hosts:
    - api.tavily.com
    - api.search.brave.com
    - api.exa.ai
  ports:
    - number: 443
      name: https
      protocol: HTTPS
  location: MESH_EXTERNAL
  resolution: DNS

templates/kiali-httproute.yaml

HTTPRoute exposing the Kiali mesh UI through the shared gateway.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: {{ .Release.Namespace }}-http-route
  namespace: {{ .Release.Namespace }}
  labels:
    {{- include "istio.labels" . | nindent 4 }}
  annotations:
    link.argocd.argoproj.io/external-link: "https://kiali.opi5cluster.co.uk"
spec:
  parentRefs:
    - group: gateway.networking.k8s.io
      kind: Gateway
      name: istio-gateway
      namespace: istio
      sectionName: websecure-uk
  hostnames:
    - kiali.opi5cluster.co.uk
  rules:
    - backendRefs:
        - name: kiali
          port: {{ .Values.kiali.server.port }}

• Traefik (decommissioned 2026-09-20) still has its own page under Decommissioned Services.
• All five upstream charts (base, istiod, cni, ztunnel, gateway) pin the same 1.30.4 version and move together; kiali-server is pinned independently.
• Ambient enrollment is per-namespace via the istio.io/dataplane-mode=ambient label; infra namespaces (storage, DR, observability, GitOps) stay out deliberately.

Trade-offs

← Back to Platform & Infrastructure · All service groups