Reclamation Configuration Reference
The configuration surface for projection-cell reclamation and origin retirement. Reclamation and retirement are on by default. Deletion itself is always available and always converges regardless of these settings — they only tune when the space a deleted or superseded value occupied is physically reclaimed.
Concept: Deletion and Reclamation →
Enabling Reclamation
Whether the reclamation scheduler ticks. Deletion (bondy_db:delete/3) always converges and is unaffected by this setting: turning reclamation off only stops the underlying tombstone from being physically reclaimed once every node has certified it causally stable. With it off, tombstones are retained indefinitely, exactly as if reclamation did not exist.
Interval between reclamation passes. Deliberately much larger than the compaction cadence: reclamation is a space concern, not a liveness one, and each pass re-derives stability from peer state that only changes as anti-entropy rounds complete. 0 disables periodic passes — an operator can still trigger one explicitly.
Cells scanned per pass batch. The sweep runs inside the applier — the one process on each node that writes to that node's local projection for a shard — so this bound caps how long one batch can stall a concurrent apply. A pass loops batches to completion; writes interleave between batches.
Origin Retirement
Whether the origin-retirement pass runs. When enabled, each node reacts to an observed membership removal — and reconciles once at boot, covering removals that happened while it was down — by forgetting departed peers from its peer state and reaping dead origins from cell states by complement. Each pass also pulls every member's retirement set and unions it in, and drops the applied-frontier entry of any origin every member has retired.
The pass is fail-closed: if any current member cannot be queried, nothing is learned and nothing is reaped, and the pass retries on the next trigger. It never retires an origin on its own — see Retiring a decommissioned node.
Where this node stores its retirement set, relative to platform_data_dir unless absolute. A retirement is permanent, so it has to survive a restart: a node that forgot one would re-learn the origin's frontier entry from a peer and then read its own absence as a deficit, paying a catalogue re-bootstrap for data it already holds. A node that cannot write this file refuses to retire and refuses to reap — retirement is disabled there and nothing else is affected.
The file is written before the retirement takes effect, so a failed write retires nothing at all — there is no half-applied state to recover from, and the operation can simply be retried. A node whose write fails keeps its already-persisted retirements and stops reaping until a write succeeds; the next successful write restores both.
Because reaping requires every member to hold the retirement, one node that cannot persist stops the whole cluster from reclaiming. A configured path that cannot be read or written therefore raises the bondy_oplog_retirement_not_persistent alarm as well as logging at error, once per episode. An unconfigured path is a deployment choice, announced once at info and never alarmed.
Interval between periodic retirement passes, in addition to the membership-event trigger. The periodic pass covers origin-epoch turnover that produces no membership event — a node that loses its storage and rejoins under the same name mints fresh origins without any member joining or leaving, and only a periodic pass on the surviving nodes reaps the dead epoch. It is also the cadence at which retirements propagate between nodes.
Retiring a Decommissioned Node
Removing a node from the cluster membership stops it participating; it does not release the applied-frontier entries its origins hold on every surviving replica. Those entries are permanent claims — one per durable shard, so the cost of a departure scales with db.main.shard_count — and only a retirement releases them.
Retirement is an operator act, never derived from membership, because membership is reversible: a departed node returns with the disk it left with and resumes minting under the same origin, and survivors that had reaped it on absence alone would silently skip its new events.
After removing the node from the membership, on any remaining node:
{ok, Retired} = bondy_oplog_origin_retirement:retire_dead().This takes the origins no current member claims and retires them. Each retirement is banned, persisted, and replicated to the other nodes by the periodic pass; once every member holds it, each node drops the corresponding frontier entries on its own next pass.
Retire only after convergence
Retiring bans the origin cluster-wide, and a ban is permanent. Any of the departed node's events a replica has not yet applied become permanently unreachable to it. A reap taken over replicas that disagree on the origin's sequence logs a warning and emits [bondy_oplog, retirement, reaped_unconverged], which is the signal that retirement happened too early.
A partitioned node cannot retire the rest of the cluster: membership changes only by a deliberate join or leave, so an unreachable peer is still a member, and a member that cannot answer aborts the pass.
Run it with every instance up. The set of unclaimed origins is built from what nodes advertise, and a durable instance that is stopped does not advertise its origin, so a live origin could be caught in it. This node's own half is checked — while any local instance is down the call refuses with {error, {instances_down, Ids}} — but a peer under-advertising cannot be detected, so the instruction stands as well as the check. The origins retired are named in a notice log, but retirement cannot be undone.
Shared Scheduler Machinery
The reclamation scheduler is an instance of the same scheduler that drives compaction. One option is shared between them.
Cap on concurrently running trigger workers, per scheduler instance — shared between the compaction scheduler and the reclamation scheduler. This is a distinct subsystem from db.gc_interval/db.gc_heap_delta, the per-instance BEAM heap monitor that happens to share the "gc" name; see Instance Memory Management.
Compaction and reclamation both wait for every cluster member, on every database, the in-memory registry included: a silent member holds them down until it leaves the cluster through bondy.cluster.leave, and no timeout changes that. With cluster.enabled = off there are no members to wait for, and both run on the node's own history.
Telemetry
Reclamation fails silently in both directions, so its observability surface is part of the contract.
| Event | Emitted | Measurements | Metadata |
|---|---|---|---|
[bondy_oplog, applier, cells_swept] | Per sweep batch | scanned, discarded, reduction_skipped, skipped | instance_id, stable_hlc |
[bondy_oplog, reclamation, stalled] | Every attempt that reclaims nothing | count | instance_id, reason (idle, unconfirmed, no_members, membership_unavailable, no_frontier, unservable, non_event_frontier, no_applier), missing_members |
[bondy_oplog, scheduler, gc, trigger_outcome] | Per trigger run, every scheduler instance | count | scheduler, instance_id, outcome |
[bondy_oplog, retirement, completed] | Per successful retirement pass | dead_origins, origins_reaped, origins_scanned, retirements_learned, frontiers_reaped | forgotten_peers |
[bondy_oplog, retirement, skipped] | Per pass whose reap-by-complement aborted; replication runs either way | retirements_learned, frontiers_reaped | reason |
[bondy_oplog, retirement, reaped_unconverged] | A frontier entry dropped while members still disagree on the origin's sequence | count | instance_id |
[bondy_oplog, applier, origins_reaped] | When a reap pass rewrites cells | cells, origins | instance_id |
[bondy_oplog, applier, retired_pairs_dropped] | A page merge carried events authored by a retired origin | count | instance_id |
idle means that this node has applied nothing on the shard, so it holds no cell to reclaim. Nothing is stalled. A shard whose tree compaction has emptied is not idle. idle is the only reason that never produces a warning log; see Deletion and Reclamation for the full reasoning behind that distinction.
Alongside the telemetry, a scheduler-driven reclamation stall also produces the "GC stalled" warning log. The warning names the instance, the reason, the missing members and each member's sync recency (member_status). Each scheduler writes at most one line per instance per 60 seconds. The compaction scheduler writes the same warning for a held compaction, and the scheduler field names the scheduler. The telemetry is not rate-limited.
See also
- Deletion and Reclamation — the model these options control.
- Data Storage & Active Anti-entropy Configuration Reference — sharding, pack-store durability, the write-ahead log, and anti-entropy sync.