The data-safety model: why a retained backup can never lose its bytes

This page is the trust argument an evaluator reads before deciding whether to rely on downpipes. It answers one question precisely: across the whole product, the only write path that ever removes archive bytes is retention pruning, so what stops that path from deleting something a backup you are keeping still needs?

The short answer is that deletion is driven by what the manifests say is referenced, never by how old an object is, and the design fails safe at every step where a read could go wrong. This page works through each part of that claim against the code that implements it. For the operator-facing knobs (keepRuns, keepDays, the enforce gate), see retention and pruning; this page is the why-it-is-safe complement to that how-to.

A note on language before the detail. The archive is tamper-evident, not unbreakable: a rewrite or a rollback is detected and surfaced loudly with a non-zero exit, which is a different and more honest claim than promising it can never happen. That distinction runs through everything below.

The core invariant: deletion is manifest-driven, never age-driven

A segment object (seg/<aa>/<segId>.seg) is deletable only when no retained run’s shard manifests reference its content id. The format forbids any object-store lifecycle or expiry rule on the seg/ and run/ trees, because a rule that expired objects by age would silently break dedup: a content-addressed segment is shared across runs, so an object that looks unreferenced may be the only copy a run you are keeping still depends on. The engine configures no such rule, and a deployment that adds one is misconfigured.

So there is no time-to-live, no bucket lifecycle policy, and no listing or age heuristic anywhere in the delete path. The planner works entirely from the decrypted manifests of the runs themselves.

How the orphan set is computed

The planner partitions a downpipe’s currently-active runs into the ones the policy retains and the ones it supersedes, then reads every run’s decrypted shard manifests and forms two sets of segment object keys.

A segment referenced by any retained run is in retainedSegRefs and is therefore subtracted out of orphanSegs, so it is never a delete candidate. A segment is deleted only when it appears in a superseded run and in no retained run at all. Content addressing makes a shared segment the same object key in both runs, because the segment id is the hash of the addressed content under the per-downpipe key, so a value unchanged across runs dedups to one object that the set-membership test sees in both groups and keeps. The set difference is over exact object keys, which is why there is no heuristic and no age comparison: an unreferenced segment is unreferenced by construction.

The abstain invariant: a transient read failure never deletes

The subtraction above is only safe if the protected set is complete. If a retained run could not be read, its segment references would be missing from retainedSegRefs, and a segment shared between that unreadable retained run and a superseded run would fall into orphanSegs and be deleted, breaking the run you meant to keep. A missing protected reference does not shrink the orphan set, it grows it, which is the dangerous direction.

So the planner does the cautious thing. If any retained run fails to enumerate on a pass, the whole prune pass abstains: it supersedes nothing, deletes nothing, records why it deferred, and retries on a later tick when the run is readable again. The planner that opens each run uses the same in-account read-back path the drill uses, and that open verifies the whole chain as a side effect, so a tampered or missing run throws rather than returning a partial set, and the abstain path catches it.

The complementary case is handled with the same care from the other direction. An unreadable superseded run is simply skipped this pass: it is not superseded, its RUNLOG entry stays active, and its run-tree and segments are left untouched, so a transient failure there cannot delete data either. Both halves mean a read that goes wrong defers work rather than destroying it.

Crash-safe ordering: supersede first, delete second

The apply path is the one place in the retention feature that writes or deletes, and the caller reaches it only when the per-downpipe enforce gate is set to the literal value true; the dry-run default never gets here. The order it writes in is the crash-safety invariant.

First, it marks the superseded runs in the RUNLOG and re-signs the whole log under the RUNLOG lock. Only then does it delete the run-tree objects, and then the orphan segments. If the process dies after the supersede commit but before or during the deletes, the RUNLOG is already consistent (the runs are superseded) and the not-yet-deleted objects are simply objects a reader reports as orphan candidates, which is safe, and a later prune finishes the deletion idempotently because deleting an absent key is a no-op and the supersede step skips entries that are already superseded. A referenced segment is never deleted, because the planner already subtracted every retained reference from the orphan set before the apply began.

Supersede is not remove

The engine’s own prune marks a run’s RUNLOG entry status="superseded" in place and re-signs the whole log. It never removes an entry. This is a deliberate format rule, not an implementation detail: the writer side that supersedes entries closes the contract both readers already honour, where a superseded entry is retained and is not a rollback, while a missing entry is.

The offline downpipe prune cannot do the marking half, and it is worth being exact about why rather than leaving the difference implied. Re-signing the RUNLOG needs the signer private key, which is the engine’s; the offline tool is given only the signer public key so that it can verify what it reads and forge nothing. So an offline prune deletes the run’s objects and leaves its entry reading status="active".

That state is safe, and it is legible. Nothing breaks: the entry is still present, so the chain is intact and the anti-rollback rule below is unaffected, exactly as for an engine prune. What it costs is one distinction. A run listed in the RUNLOG whose tree is entirely gone is what an offline prune leaves behind, and it is also what deleting that run would look like, and no reader can tell the two apart because the record that would have distinguished them is the one the offline tool cannot sign. The reader says so plainly when you open such a run rather than reporting a missing file, and downpipe keys --which counts them for you, so comparing that count against the retention you actually ran is the check that closes the gap.

This distinction is specific to the standalone offline tool, not to holding a break-glass-only posture. When you prune from the console instead (the break-glass prune panel, covered on retention and pruning), your browser supplies only the recovered per-run keys, and the ENGINE plans and applies the delete with its own signer private key, exactly as the scheduled pass would with a held key. So a console-driven prune marks and re-signs the RUNLOG properly, with no status="active" gap to close; the gap above is only what you get by running downpipe prune on its own, with no engine in the loop at all.

The reason is the anti-rollback check covered next. Removing a RUNLOG entry would break the per-downpipe chain and read as a rollback, so the prune keeps every entry and only changes its status. A reader walking the chain backward may reach an entry whose run/<runId>/ tree has been pruned away; that is the expected after-prune state and is explicitly not a rollback, because the chain anomalies are checked against the RUNLOG entries themselves, which retain the pruned entries marked superseded, never against whether each previous run’s tree is still present.

The anti-rollback reader rule, precisely

The anomaly detection is shared between the TypeScript engine reader and the Go offline reader, because the in-bucket chain is something the reader is the authoritative verifier for. A signed RUNLOG can still be internally chain-anomalous, and both readers run that same detection and reject three specific shapes as a rollback. In the offline Go CLI that rollback surfaces as exit code 5; the in-Worker engine reader reports the same verdict to its caller rather than exiting a process.

The ordering matters and is the same on both sides. The checks sort the entries by index first, because line order is not load-bearing: indices are allocated at trigger time but entries land at finalise time, so concurrent runs legitimately append out of allocation order, and the whole-document signature is the integrity anchor that makes reordering without re-signing impossible. The one index fact that is a corruption signal is a duplicate, which is why a gap is fine but a repeated index is not.

The rule above assumes the RUNLOG can be read at all

Every row in that table is a verdict reached by inspecting a document the reader has in hand. There is a second class of outcome, and it is the one an adversary would reach for first, because deleting a document is easier than forging one. The RUNLOG or its detached signature can be absent, unreadable, unparseable or empty; the signature can fail to verify against your pinned signer; or the log can be well-formed and simply not carry the run you asked for, or carry it in a form that does not bind to the signed root manifest.

None of those is a rollback finding. Each is the absence of a finding, and both readers now say so with a field of their own, separate from the rollback signal: a check that ran and found something is a finding, a check that could not run is an unknown, and an operator deciding whether to restore needs to be told which one they are holding. Both refuse either way. In the offline CLI both are exit code 5, and the stderr line names which one it is; the acknowledgement that reaches an unrunnable check is --allow-unverified-runlog rather than --allow-stale, which is an age word and refuses a second time here.

This was for a period the weaker half of the product, and stating it plainly is more useful than presenting the fix as though it had always been so. Two things were wrong at once and both were corrected on 5 August 2026. The engine’s copy of the check shared one failure helper across those early returns, and that helper reported no rollback, so a forged _RECOVERY/RUNLOG.sig and a deleted RUNLOG both opened an archive that a benign below-pin index would have refused. On the offline side the refusal was real, but --allow-stale waived it: an operator who typed a word meaning “I know this run is old” waived a signature check with it. An unreadable or unverifiable RUNLOG now refuses in-account, and offline it takes the acknowledgement that names what it waives.

There is also a layer the in-document checks deliberately do not cover. Substituting an older but validly signed RUNLOG and its signature wholesale (a tail rollback) is caught only by an out-of-band minimum-index pin, not by any in-document check. The division of labour is explicit: the whole-document signature catches a rewrite of the stored bytes, the chain anomalies catch a removal or rewrite within an otherwise-signed document, and the out-of-band pin catches a wholesale tail rollback.

The evidence a detection leaves behind

A rollback or unreadable-RUNLOG verdict does not arrive carrying the document, and it cannot: the reason string the detector builds names your own run and downpipe ids. What it does leave is a forensic row in your support pack, written by the engine into your own account and readable by you. The row names which of a closed set of ten recorded kinds matched, which is a wider vocabulary than the rollback shapes in the table above because it also covers the unreadable and unverifiable states in this section. It carries the pair of disagreeing indices the detector compared, how long the chain was, and a twelve-character one-way digest of the RUNLOG bytes that you can reproduce by hashing the object in your own bucket. For the kinds, the read-fault classes beside them, and how to check the digest yourself, see the support bundle.

The honest framing: tamper-evident, with one delete-refusing posture

Everything above detects and refuses a rollback rather than making one impossible at the storage layer. That is the honest claim: the archive is tamper-evident, and a rollback surfaces as exit code 5 rather than passing silently. The genuinely delete-refusing posture is a separate storage choice, and there the trade is different: on an object-lock or WORM destination the bucket itself refuses deletes, so retention pruning honestly cannot run at all and history simply accumulates.

Where pruning is deferred, and why that is safe

Two configurations stop the prune entirely. In both, backups keep working and history accumulates; only the deletion side is held back, and each is a deliberate trade rather than a gap.

The break-glass-only deferral mirrors the same honest skip the in-account drill makes when there is no read-back key. In neither posture is anything at risk: the worst case is that old history is kept longer than the policy asked for, which is the safe direction for a backup product.

Where this fits

Last updated 5 August 2026.