Skip to content
downpipes docs

Remove a destination without orphaning your backups

Removing a destination is owner-only, and the engine will refuse it in two different situations. The two look similar in the moment and are not the same: one you can override, and one you cannot.

Refusal one: a downpipe pins it

If any downpipe names this destination explicitly, as its primary or as a replica, the removal is refused and there is no force for this one. The message names the downpipes.

The reason it is absolute is that a pinned destination id never silently falls back to something else. Removing it would leave those downpipes pointing at an id that does not exist. A downpipe with no other destination would fail its next run rather than quietly write somewhere else. A fan-out downpipe would skip the missing id and keep fewer copies than you configured. Reassign them first, then remove.

Refusal two: it holds the only proven copy

This is the data-loss guard, and it is the one force overrides.

Under 3-2-1 failover a run may have sealed here and not yet been replicated anywhere else. This destination is that run’s recorded origin, and for the moment it is the only proven copy.

A downpipe’s runs stop counting against this guard only once another destination’s proven window contains all of them at both ends. Catching up is not enough on its own: that destination’s replication must have reached the newest of those runs (its holdsIndex), and its proven floor, the lowest run index it has contiguously observed (holdsFrom), must sit at or below the oldest. A destination with no recorded floor contributes no coverage at all, so the removal stays blocked. That is deliberately the safe direction: a replica that only started observing at the history ring’s floor cannot prove it holds a run from below that floor, and the guard refuses to accept a copy it cannot prove.

The refusal names the count and the downpipes: the only proven copy of N backed-up runs.

The guard reads only the runs still in each live downpipe’s run history, which keeps at most 50 rows per downpipe. An older run, or a run of a deleted downpipe, does not count.

Unpinned downpipes count too. A downpipe with no explicit destination seals to whatever the default is and records no origin id, so removing the sole default still counts those runs against the guard. You do not have to have pinned anything to be protected by it.

What forcing actually costs

Forcing does not merely un-reference those archives. The destination’s stored credential goes with it, so the engine can no longer address the bytes. Here is what that does and does not mean. Nothing in the bucket is deleted: removal filters the entry out of the stored destination collection and touches no object. What is destroyed is downpipes’ copy of the credential, and a restore of an affected run then fails with the engine’s own terminal reason, which names the remedy: this run’s only copy was on a removed destination; re-add it (or a destination that holds these bytes) and retry, or re-point the restore with an explicit destination (REASON_ORIGIN_REMOVED, engine/src/restore-reasons.ts).

So the recoverable case and the unrecoverable one turn on where the access key lives. If you still hold it in your own provider console, you can re-add a destination pointed at the same endpoint and bucket. The re-added destination gets a new id, so the restore must name it as an explicit destination. If the only copy of that key was the one downpipes held, the bytes are stranded for good.

So the guard is not being cautious about tidiness. It is the difference between N runs you can still restore and N runs you cannot.

Forcing is still allowed, because sometimes dropping those copies is the deliberate decision. When you do, the engine records how many runs it just orphaned, computed before the removal and kept as the magnitude of the loss. That count exists so the event is auditable later rather than being reconstructed from what someone remembers choosing.

Usually the right move is to wait

The replication pass copies the backlog every tick for each downpipe with two or more destinations. For those downpipes this guard clears itself. A downpipe’s runs leave the count once another destination holds all of them, and then the removal succeeds with no force at all. Replication skips a downpipe with one destination, so waiting does not clear its runs. The refusal tells you when this applies.

Three options, in the order worth trying them:

  1. Wait. Watch the redundancy map’s Copies row, which shows how many destinations hold each fan-out downpipe’s latest successful run. When the runs this guard names have a second copy, the refusal stops.
  2. Reassign the downpipes to another destination, so new runs seal elsewhere.
  3. Force, having decided those copies are expendable.

The default is promoted silently

One side effect worth knowing before removing anything: if you remove the current default, the first remaining destination in the list silently becomes the default. It is the first one, not the one that sat next to the one you removed: the engine filters the removed entry out and takes position zero of what is left (removeDest, engine/src/sched/scheduler-do-dest-config.ts). Nothing asks you to confirm which one, and it is chosen by position rather than by any preference of yours.

So after removing a default, check which destination is now the default before the next run seals to it. Every unpinned downpipe follows that choice.

What is recorded

A removal the engine accepts, or queues for a second owner’s approval, raises a destination-change alert. The alert says explicitly when force was used. Your notifications can then tell a forced drop from an ordinary one, not only the engine’s own records.

The two refusals are recorded as separate closed classes, in-use and orphan-guard, so “a downpipe still names this destination” and “the engine refused to drop the only proven copy” are distinguishable after the fact. Both classes are engine refusals. An operator who reads a refusal and walks away leaves no record of its own: what is recorded is the refusal, not the decision that followed it. A forced removal, by contrast, is recorded, with the count of runs it orphaned and up to five of their downpipe names.

Where this fits

Last updated .