Deploying on Kubernetes

On Kubernetes, Nova is one helm install: the chart brings up the controller (a Kubernetes operator) and the Constellation it maintains: the serving fleet of stars, the audit pool, the oxia metadata plane, the object cache, and the alerting surface. Everything about the system is a field on the Constellation spec; the chart's values are that spec, verbatim.

The two loops

One line governs the design: Nova decides where streams live; the controller decides the shape of the fleet. Nova's own placement and rebalance (fast, per-stream, reading live load) stay inside the stars. The controller owns the slow structural loop: how many stars exist, in which locations, their identities, upgrades, and departures. It acts only through the node-scoped operator verbs (Drain, Decommission) and never names a stream.

The loops meet through the signal plane: every star publishes its stats (utilization over the CPU and network axes, the stranded bit, observed metastore p99, a ring histogram of its metadata operations) into the cluster overview, and the controller reads one witness star's overview per pass, never a per-node scrape.

A second boundary: the controller scales pods, never machines. It holds no cloud credentials. Machines follow through pod pressure: a node provisioner (Karpenter, cluster-autoscaler, static pools in dev) reads Pending pods' requests and zone affinities and mints what they describe. Buckets and IAM are equally outside its charter: storage is referenced and validated, never provisioned.

Install

constellation:
  spec:
    image: ghcr.io/supabase/novad:2.0.0
    serving:
      replicas: 3
      locations: [eu-west-1a, eu-west-1b, eu-west-1c]
    storage:
      bucket: nova-prod
      serviceAccountName: novad   # IRSA-annotated, created by you
    observability: { rules: true, monitors: true, dashboards: true }

# Optional: the internet-facing tier.
gateway:
  spec:
    image: ghcr.io/supabase/novagateway:2.0.0
    replicas: 2

helm install nova deploy/helm/nova \
  --namespace nova --create-namespace \
  --values my-values.yaml

kubectl -n nova get constellation nova -o jsonpath='{.status.stars}'       # the ledger
kubectl -n nova get constellation nova -o jsonpath='{.status.conditions}'
kubectl -n nova get pods

The Constellation spec

The CRD is the documented surface (kubectl explain constellation.spec). The blocks, with the defaults an empty spec renders:

apiVersion: nova.dev/v1alpha1
kind: Constellation
metadata: { name: prod, namespace: nova }
spec:
  image: ghcr.io/you/novad:1.4.0
  serving:
    replicas: 3
    locations: [az-1, az-2, az-3]
    reapAfter: 10m
    autoscale: { min: 3, max: 21, targetPercent: 60,
                 lowWaterPercent: 30, scaleDownAfter: 30m }
    slice:
      netCapacity: 750M            # wire bytes/sec → --net-capacity
      cpu: "2"                     # request; cpu is never limited
      memory: { request: 4Gi, limit: 8Gi }
    nodeSelector: { nova.dev/pool: serving }
  storage:
    bucket: nova-prod
    credentialsSecretRef: s3-creds # omit for the default chain (IRSA)
    express: { az-1: bx-1, az-2: bx-2, az-3: bx-3 }
  auth:
    mode: jwt                      # the controller mints and owns the key set
  metadataPlane:
    oxia:
      servers: 3
      streamsPerShard: 10000
      storage: 16Gi
      walStorage: 8Gi
      autoscale: { maxServers: 5, opsPerShard: 2000, leadersPerServer: 7,
                   p99ThresholdMs: 50, p99AlarmAfter: 5m }
      resources:   { cpu: 4500m, memory: 4Gi }   # requests = limits
      coordinator: { cpu: 500m, memory: 256Mi }
      nodeSelector: { nova.dev/pool: plane }
  audit:
    replicas: 2
    autoscale: { min: 2, max: 8, ageThreshold: 5m, busyPercent: 50 }
    slice: { netCapacity: 438M, cpu: "4",
             memory: { request: 8Gi, limit: 12Gi }, scratch: 300Gi }
    s3Budget: 200                  # reclamation dials: size so settle
    rewriteMiB: 256                # sustains ingest
    nodeSelector: { nova.dev/pool: audit }
  objectCache:                     # absent = direct reads
    instances: 2
    nodeSelector: { nova.dev/pool: cache }
  upgrade: { width: 4, canaryWindow: 5m }
  observability: { rules: true, monitors: true, dashboards: true }

storage also takes endpoint, region, and pathStyle for non-AWS stores; objectCache takes image, slice, dnsServers, and metricsEndpoint; every component takes tolerations beside its nodeSelector.

Identity: the ledger

Stars hold no data on local disk, so nothing about a star's identity lives in a volume. The controller mints each StarID, injects it as --node-id, and records it in the Constellation's status: the ledger, one entry per star with its pod name, StarID, location, image, template hash, and ladder phase. Each entry renders one bare pod (no StatefulSet, no PVC). A restarting pod resumes its star; a vanished pod is recreated under its standing identity next pass. Pods are create-never-mutate: changing a star's substance (image or settings alike) mints a replacement identity and walks the old one out. New stars land in the least-populated location among serving.locations, enforced twice: --location tells Nova, a required node affinity on topology.kubernetes.io/zone tells Kubernetes.

The slice model

Kubernetes cannot schedule the axis that binds this workload (network bandwidth), so the controller packs machines by the slice: one pod's declared share of net capacity (wire bytes), cpu, memory, and, for auditors, scratch. Requests equal the slice, so pods-per-machine is scheduling arithmetic; --net-capacity is stamped from the slice's net figure, so the saturation axis divides by a share that is physically true. Wire bytes, not client bytes: append, read, and express fanout all cross the NIC, so wire is roughly 2 to 3× client bytes. Stars spread across machines by a maxSkew-1 hostname topology spread: co-residence is legal, imbalance is not.

Auditor scratch is real to the scheduler: an ephemeral-storage request plus a size-limited scratch volume, with the fetch-cache budget derived at 90% so admission backpressure, never kubelet eviction, is the valve. audit.slice.scratch: 0 renders no scratch and no fetch cache: the auditor keeps its ranged-GET strategy and the object cache absorbs the amplification. Oxia servers run requests = limits: every ref commit is a quorum write through them, and a quorum server must not steal. Every figure is provisional pending calibration.

Sizing loops

Two disciplines run through every loop. Only a settled world is judged: every pod ready, no departure mid-flight. And the starved/wedged split: a high signal with busy workers is a capacity problem the controller fixes; a high signal with idle workers is a fault it freezes on and surfaces as a condition, never papering over with money.

Serving fleet. desired = ceil(N × avgUtil / target), target 60%, deliberately below the rebalance gate's 85%, so the gap is spike-absorption headroom. Any star reporting stranded (saturated, and no live peer fits even one of its streams) adds capacity at once. Stranded persisting after new capacity registers convicts an indivisible hot stream instead: a condition, never a scale-up. Shrink is asymmetric: only when the whole fleet's peak sits below 30% for 30 minutes is exactly one star released, through the ladder.

Audit pool. Signal: backlog age, never rate. Due-queue age past ageThreshold with a busy pool is starvation: one worker joins. Age with an idle pool is a wedge: the pool freezes and AuditWedged goes true. Auditors are fungible; they quiesce on SIGTERM and shrink by deletion, and the ring re-deals duty shards among survivors.

Metadata plane. The plane scales by heat. Every star and auditor publishes a 4096-bucket histogram of its metadata operations over oxia's hash ring; the controller merges them and slices by the shard ranges in the coordinator's status ConfigMap. The hottest shard sustaining above 60% of opsPerShard for ten minutes splits at the load-weighted median; a shard whose single hottest bucket carries most of its load is bracketed onto a private shard, and a private shard still hot raises MetadataHotStream. Servers follow shards (ceil(shards / leadersPerServer), clamped to [servers, maxServers]). Splits are forever and servers grow only. p99 is a pure alarm (MetastoreDegraded); it moves nothing.

Settings

The controller renders a closed flag set. Two doors reach past it:

• Dials. audit.s3Budget (object-store requests/sec) and audit.rewriteMiB (MiB/sec) are first-class fields; every real deployment must size them so settle sustains ingest.
• Env. serving.env and audit.env pass environment variables to the containers; the daemon reads a NOVAD_* twin for every flag, and its precedence (flag > env > config file) keeps rendered flags above anything set here. A name whose flag the controller renders is refused loudly.

Every ledger entry carries a hash of its rendered template (image, args, env, resources, scheduling). An entry whose hash differs from the current render is drifted, and rolls exactly like a stale image. Auditors churn instead: one drifted pod per pass is deleted once the rest of the pool stands ready.

Upgrades and departures

Change spec.image; the roll is surge-then-drain. A ledger entry whose template differs from spec is blue; its green replacement is minted at the blue's own location, and the blue queues for the ladder. Capacity never dips below N: the newcomer warms up on the drained star's handoffs. The first green is the canary, judged alone through upgrade.canaryWindow: it must register on the overview, and no star anywhere may be stranded. A dirty canary sets UpgradeHalted and the roll holds where it stands. Past the canary, upgrade.width replacements run in flight, capped at a quarter of the fleet. Rollback is not a mechanism: revert spec.image and the same machinery runs backwards.

Departures walk the ladder (drain, the terminate gate, decommission, delete) one star per pass; a rung that cannot advance is held with a named cause (DepartureHeld), never failed. Unplanned losses are reaped through a witness after reapAfter. Node lifecycle has the full state machine.

The Gateway and the door

The tier itself (what it holds, its middleware, plan-driven serving) is the Gateway architecture page.

The Gateway resource is separate because it has its own release cadence: upgrading the fleet never touches it. It references its Constellation and renders a Deployment (per location when an object cache is present):

apiVersion: nova.dev/v1alpha1
kind: Gateway
metadata: { name: edge, namespace: nova }
spec:
  constellationRef: prod
  image: ghcr.io/you/novagateway:1.4.0
  replicas: 2
  resources: { cpu: "1", memory: 1Gi }
  ingress:
    className: alb
    host: nova.example.com
    annotations:
      alb.ingress.kubernetes.io/scheme: internet-facing
      alb.ingress.kubernetes.io/target-type: ip
      alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:...
      alb.ingress.kubernetes.io/load-balancer-attributes: idle_timeout.timeout_seconds=30
  # tlsSecretRef: edge-tls — self-managed TLS at the gateway instead

ingress is cloud-neutral (class and annotations verbatim); the ALB posture terminates TLS at the load balancer, and the 30-second idle timeout is safe against the SSE path's 15-second heartbeats. Absent, the tier stays cluster-internal.

Storage credentials

Leave credentialsSecretRef unset and name an IRSA-annotated service account in storage.serviceAccountName: novad falls through to the AWS SDK default chain and picks up the web-identity token the role-annotated account injects. Scope the role to the bucket (and, for Express, grant s3express:CreateSession on the directory buckets). credentialsSecretRef names a Secret holding static keys for non-AWS stores.

Conditions and observability

Alert on AuditWedged (reclamation behind on age with an idle pool: a fault) and UpgradeHalted (a dirty canary). DepartureHeld names a rung waiting on convergence; MetastoreDegraded and MetadataHotStream come from the plane loop. With observability on, the controller ships and repairs the PrometheusRule (fleet and node saturation, stranded, per-class write SLOs, writer-limbo age, rewrite-backlog age: age, never rate), the ServiceMonitor/PodMonitor pair, and the Grafana dashboards as sidecar-labeled ConfigMaps. Prometheus Operator CRDs are required for rules and monitors.

The Pulumi reference deployment

deploy/pulumi/ provisions the cloud half on AWS and installs the chart: a VPC across three AZs, an EKS cluster with node pools labeled nova.dev/pool: serving|audit|plane (plus optional bench and gateway pools), the S3 bucket with an IRSA role scoped to it, S3 Express directory buckets per expressZoneIds, ECR repositories the deploy script fills (nodes never pull from Docker Hub), and kube-prometheus-stack on a tainted observability node.

cd deploy/pulumi
./deploy.sh --stack dev

The script is idempotent: it builds the image for linux/amd64 and linux/arm64, mirrors oxia into ECR, and runs pulumi up. Exposing the Gateway is a deliberate two-step door (request the ACM certificate and publish its validation CNAME, then arm the ingress once the certificate is issued), so pulumi up never hangs on someone else's DNS. Every knob lives in Pulumi.yaml's config: block.

Source documents

• deploy/helm/nova/README.md, deploy/helm/nova/values.yaml, deploy/helm/nova/crds/
• operator/README.md
• docs/design/023-constellation-controller.md
• deploy/pulumi/README.md