Due queues

The auditor never looks for work; work announces itself. The foreground arms a due marker in the same commit that creates the condition, and discovery is a bounded, ordered range read of the markers a member owns. This page is the shape of that keyspace, the invariant that keeps every object watched, and the dispatch pass that turns ripe markers into leased jobs.

Work announces itself

Every condition that needs background work is created by a metastore write, and the marker for it rides that write. A span commit arms the batch marker for its slots; a stream's first commit arms its retention marker; a deletion arms its teardown marker; a chain mint re-announces the star's unsettled epochs for settlement; a killer of a span arms the object's reap-check before the kill; every publish arms its orphan intent before the PUT. The auditor's cost is proportional to the work that exists, never to the number of streams, objects, or chains.

The due keyspace

Markers live in the work namespace under due/{queue}/{shard}/{order}~{subject}, where the subject is what the marker names: a stream, an object key, or a chain. The shard is a hash of the subject over the frozen shard count (64); the order and subject fuse into one final segment so every marker is a direct child of its shard prefix and one ordered scan walks it. The shard is also the locality key, so a shard's markers colocate on one metastore shard.

Object keys embed their slashes folded to !, which no object key may contain. Object-scoped markers (reap, gc) and batch markers also write entries into a by-stream secondary index, atomically with the marker and dying with it, capped at 8 attributed streams and flagged shared beyond the cap: the admin surface's per-stream pending-work view, read as one bounded prefix scan.

Shards and seats

A member's seat is the set of shards rendezvous deals it over the ring roster: shard s goes to the live member with the highest score for it. Discovery reads only owned shards; the roster is re-read at every pass boundary, so a membership change moves discovery within one pass and transfers nothing.

The denorm queue is the exception. Its markers are placed per chain, not per shard (all of a chain's markers hash into one shard by {star}~{epoch}, and the era needs them together), so every member scans every shard's schedule and the judge skips markers whose chain is placed elsewhere (registered stars are dealt one per seat in sorted order). Leases remain the mutual exclusion either way.

Ripe versus standing

A marker stands from the moment it is armed; it is ripe once its ready time has passed. How a walk treats an unripe marker depends on the queue's order:

• Time-ordered queues (teardown, settle, retention, reap, trimcheck) carry the ready time as the key's order prefix. The first unripe marker stops the walk: everything after it is younger still. A limit-1 read of the shard is therefore the oldest pending item, which is what the backlog-age gauge peeks.
• denorm orders its primary keys by chain coordinate; its schedule is a separate index keyed by ready time, and the drain walks that index. The walk stops at the first unripe entry.
• orphan orders by object key, which is no age order. Ripeness is judged per intent from its recorded birth against the orphan grace; an unripe intent is skipped and the walk continues.
• gc has no readiness. Every marker is ready; the order is reclaim size.

A malformed marker no one could ever run is resolved away at triage (without a lease, since there is no duplicate work to avoid) and logged at Warn.

Batch markers are chain coordinates

A batch marker's key is {star}~{epoch}~{firstSlot}: a position in one chain. Markers arm at the chain's cut cursor, always above everything already armed, and vanish only at resolution. So a limit-1 prefix read of a chain's due range yields the chain's lowest outstanding settle coordinate (the settle frontier) with no state beyond the queue itself. It only advances.

Two consumers read it. The trim-check advances the rewritten cursor to the earliest standing marker (or the checkpoint when none stands): every slot below it completed its rewrite. Teardown's first walk waits behind it: a deleted stream whose last slot sits at or above the frontier is still inside a settle job's territory, and the walk re-arms rather than race it, ripening the chain's owed markers to now so the wait is about one settle pass, not the age cap.

Orphan intents precede the PUT

Every object the auditor publishes (a settle output, a tail-fill revision, a GC revision) is announced by an intent under due/orphan/ before the PUT, and the intent is resolved only after a commit made the object visible. Chain slots are the one kind of object with no intent: their PUT is the commit, and their watcher is the batch marker the absorb arms before the checkpoint covers them.

The consequence is the reason no bucket listing exists: no object can exist that the system did not pre-announce, so a leak is either ref-named, intent-carrying, or marked, and each of those has a reader.

The watcher invariant

Every object always has at least one watcher: a ref that names it, its orphan intent, or a due marker. The duties keep it by two rules:

• A killer that cannot re-find its kill after its own marker resolves (retention's and teardown's walks, the tail-fill superseding a tail object) arms the object's reap-check before the visibility mutation. The worst crash outcome is a premature marker whose probe finds a live span and stands down.
• A duty whose own marker persists until it is done (the settle rewrite, GC) arms at completion, before resolving its marker.
• An intent resolves only once a ref names its object. A publish whose every range was already owned elsewhere leaves the intent standing, and the orphan duty judges it.

Resolving and re-arming

An idempotent delete of its key resolves a marker, always after the work it announced is durable: the reap resolves after its batch delete lands, the settle after the catalog commit, the walks after their kills. A duty that needs to try again later writes a new marker at a new ready time and resolves the current one; a crash between the two re-runs the old marker, which re-judges and converges.

Markers collapse rather than accumulate: keys are deterministic per (subject, ready time), so re-announcing the same condition rewrites one record. The reap-check rounds its grace up to a 30-second bucket for exactly this reason: the concurrent killers of a shared object's spans produce one marker and one probe, not one per claimant. Teardown's re-arms carry the original deletion time forward as the value, so the queue's age reports the true wait however many times the gate re-keys the schedule.

The dispatch pass

A pass is one discovery cycle, bounded at 30 seconds and re-run immediately while ready work remains:

1. Retry any lease release that leaked; re-read owned shards; refresh the chain placement roster.
2. Open one pull-iterator per (duty, shard) in priority order. The metastore's paging is the only prefetch.
3. Rotate: offer every live source one leased job per round. A source spent for the pass leaves the rotation: exhausted, stopped at an unripe marker, or failed (logged; the shard retries next pass). A duty already holding max(1, workers/2) jobs is set aside until a completion frees a slot.
4. Submission blocks while the pool is full; a freed worker admits the next job at once, so the pass is work-conserving and never tick-paced while work remains.
5. At the pass's end, spooled confirmed reaps leave in batches.

Groups. Two duties dispatch groups instead of single records. Teardown gathers up to 16 ripe markers from one source into one job (8 streams walking at a time). The settle rewrite gathers a chain's ripe markers into an era (up to --auditor-markers-per-era, bounded also by 64 MiB of marker payload per round, since a marker's slot inventory grows with stream density), and a running era keeps pulling its chain's newly ripe markers for up to half the job deadline. Inside an era, members settle --auditor-member-parallel abreast.

Failure posture. A failed job is logged at Warn and never propagated: an error would cancel the pool and starve every other queue, and one poisoned marker must cost one job run, nothing more. Group convergence is per member: a failed member's marker stands; its siblings resolve.

Leases are an optimization

Before a job runs, the member claims auditor/lease/{jobKey}: a session-ephemeral record that dies with the process. A claim that fails means a peer is on it: skip, never wait. Losing a claim is the mechanism working; correctness never depends on one, because every mutation is CAS-guarded and idempotent. The lease's TTL is the job deadline (10 minutes), so a crashed or wedged holder frees its job within one deadline; a live process retries leaked releases at every pass.

Batch markers carry no per-marker lease. The era's exclusion is coarser (one chain lease per running era) because marker leases would add an ephemeral write and delete per marker per era, the drain's single largest metastore write source, while dedup only needs the chain.

nova.auditor.lease.skips counts lost claims per queue; a skip against a marker more than 5 minutes past due escalates to a Warn naming the job, since a lease contended that long means the holder is wedged, not working.

What backlog age means

nova.auditor.due.age is, per time-ordered queue, how far past its ready time the oldest pending marker sits, read as one bounded peek per owned shard. It is the alerting axis, and it means something different per queue:

The gauge reports nothing for an empty queue and nothing from a dead auditor, so an alert on it needs an absent-signal companion. An unripe marker is not backlog: under an age cap a standing marker is the policy working. Rates (armed per second, resolved per second) are sizing signals, never alerts. See observability.

Source documents

• docs/adr/0001-due-queue-auditor-discovery.md, docs/adr/0003-keyspace-partitioning-namespaces.md
• docs/adr/0028-auditor-dispatch-pass.md
• internal/metadata/audit.go, internal/metadata/dueindex.go, internal/metadata/denormmarker.go, internal/metadata/reapcheck.go
• internal/auditor/dispatch.go, internal/auditor/leases.go, internal/coord/ring.go, internal/coord/claims.go