RustFS

S3-compatible object storage that backs Velero, AutoMQ, and the media pipeline

There are two RustFS instances. This one runs in-cluster and holds live data: AutoMQ segments, application buckets, and Velero staging. A second, standalone instance runs on backup-raspi3 and holds the Velero backups themselves.

ArgoCD configuration

Excerpt from argocd-apps/values.yaml in the argocd-apps chart, with annotations added for this site: 

rustfs:
  application: true
  project: cluster-services
  # false: this instance holds live AutoMQ segments and app buckets
  autoSync: false

Chart values

The full rustfs/values.yaml from the service's own chart: 

#### See Rustfs
image: registry.opi5cluster.co.uk/rustfs/rustfs:v1.0.0-rc.5

## Internal port number for Rustfs S3 API container
apiPort: 9000

## Internal port number for Rustfs Browser Console container
consolePort: 9001


persistence:
  size: 600Gi
  storageClass: longhorn-ssd-large
  accessModes: ReadWriteOnce
  volumeName: pvc-e9495802-6aa4-4f2a-bd6b-ac872d6748fd

Manifests & templates

templates/data-pvc.yaml

Longhorn-backed PersistentVolumeClaim for persistent data.

Show manifest 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ .Release.Namespace }}-data-pvc
  namespace: {{ .Release.Namespace }}
spec:
  accessModes:
  - {{ .Values.persistence.accessModes }}
  resources:
    requests:
      storage: {{ .Values.persistence.size }}
  storageClassName: {{ .Values.persistence.storageClass }}
  volumeName: {{ .Values.persistence.volumeName }}

templates/external-secret.yaml

ExternalSecret syncing the service credentials from Vault into the namespace.

Show manifest 
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: {{ .Release.Namespace }}-external-secret
  namespace: {{ .Release.Namespace }}
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: vault-cluster-secret-store
  target:
    name: {{ .Release.Namespace }}-secret
    creationPolicy: Owner
  data:
    - secretKey: API_ENDPOINT_URL
      remoteRef:
        key: rustfs
        property: internal-api-endpoint
    - secretKey: WEB_ENDPOINT_URL
      remoteRef:
        key: rustfs
        property: internal-console-endpoint
    - secretKey: RUSTFS_ACCESS_KEY
      remoteRef:
        key: rustfs
        property: user-name
    - secretKey: RUSTFS_SECRET_KEY
      remoteRef:
        key: rustfs
        property: user-password

templates/http-route.yaml

Gateway API HTTPRoute exposing the service through the Istio gateway under opi5cluster.co.uk.

Show manifest 

# Console — public HTTPS
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: {{ .Release.Namespace }}-console-httproute
  namespace: {{ .Release.Namespace }}
  annotations:
    link.argocd.argoproj.io/external-link: "https://{{ .Release.Namespace }}.opi5cluster.co.uk"
spec:
  parentRefs:
    - name: istio-gateway
      namespace: istio
      sectionName: websecure
  hostnames:
    - {{ .Release.Namespace }}.opi5cluster.co.uk
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: {{ .Release.Namespace }}-service
          port: {{ .Values.consolePort }}
      timeouts:
        request: 0s
        backendRequest: 0s
---
# API — internal gateway listener
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: {{ .Release.Namespace }}-api-httproute
  namespace: {{ .Release.Namespace }}
  annotations:
    link.argocd.argoproj.io/external-link: "https://api.{{ .Release.Namespace }}.opi5cluster.co.uk"
spec:
  parentRefs:
    - name: istio-gateway
      namespace: istio
      sectionName: rustfs
  hostnames:
    - api.{{ .Release.Namespace }}.opi5cluster.co.uk
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: {{ .Release.Namespace }}-service
          port: {{ .Values.apiPort }}
      timeouts:
        request: 0s
        backendRequest: 0s

templates/rollout.yaml

Argo Rollout workload: container spec, probes, resources and rollout strategy.

Show manifest 
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: {{ .Release.Namespace }}-rollout
  namespace: {{ .Release.Namespace }}
  labels:
    app: {{ .Release.Namespace }}
spec:
  replicas: 1
  strategy:
    canary:
      steps:
      - setWeight: 25
      - pause: {duration: 30s}
      - setWeight: 50
      - pause: {duration: 45s}
      - setWeight: 100
  selector:
    matchLabels:
      app: {{ .Release.Namespace }}
  template:
    metadata:
      labels:
        app: {{ .Release.Namespace }}
    spec:
      serviceAccountName: {{ .Release.Namespace }}-service-account
      automountServiceAccountToken: false
      containers:
      - name: {{ .Release.Namespace }}
        image: {{ .Values.image }}
        env:
        - name: RUSTFS_SECRET_KEY
          valueFrom:
            secretKeyRef:
              name: {{ .Release.Namespace }}-secret
              key: RUSTFS_SECRET_KEY
        - name: RUSTFS_ACCESS_KEY
          valueFrom:
            secretKeyRef:
              name: {{ .Release.Namespace }}-secret
              key: RUSTFS_ACCESS_KEY
        - name: RUSTFS_CONSOLE_ENABLE
          value: "true"
        - name: RUSTFS_ADDRESS
          value: ":{{ .Values.apiPort }}"
        ports:
        - name: {{ .Release.Namespace }}-api
          containerPort: {{ .Values.apiPort }}
          protocol: TCP
        - name: {{ .Release.Namespace }}-web
          containerPort: {{ .Values.consolePort }}
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /data
          readOnly: false
        resources: {}
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: {{ .Release.Namespace }}-data-pvc

templates/service-account.yaml

ServiceAccount the workload runs as.

Show manifest 
apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Release.Namespace }}-service-account
  namespace: {{ .Release.Namespace }}

templates/services.yaml

ClusterIP Services fronting the workload.

Show manifest 
apiVersion: v1
kind: Service
metadata:
  name: {{ .Release.Namespace }}-service
  namespace: {{ .Release.Namespace }}
spec:
  type: ClusterIP
  selector:
    app: {{ .Release.Namespace }}
  ports:
    - name: {{ .Release.Namespace }}-api
      protocol: TCP
      port: {{ .Values.apiPort }}
      targetPort: {{ .Values.apiPort }}
    - name: {{ .Release.Namespace }}-web
      protocol: TCP
      port: {{ .Values.consolePort }}
      targetPort: {{ .Values.consolePort }}
  • The backup-raspi3 instance is documented under Velero.

Trade-offs

Decision. Self-host an S3-compatible API

Alternative. MinIO or cloud object storage

Why. The S3 API is what unlocks tiered storage (AutoMQ, Velero) without egress costs, and RustFS stays light enough for arm64.

← Back to Data & Streaming · All service groups