Skip to content
downpipes docs

Remove a destination without orphaning your backups

Removing a destination is owner-only, and the engine refuses it in two different situations. The two look similar in the moment and are not the same: one you can override, and one you cannot.

Both refusals count the runs that no other destination holds yet, one run at a time. For a downpipe with two or more destinations, replication copies those runs at each cron tick, so the count falls as the copies land. Each refusal names the step that ends it (destInUseRefusal and destOrphanRefusal, engine/src/sched/dest-removal-text.ts).

The order that works

To move your backups from one destination to another and then remove the old one, do these steps in this order.

  1. On each downpipe, tick both the old destination and the new one. Each downpipe then has two destinations, and replication copies its runs.
  2. Wait. At each 15-minute cron tick, replication copies to the new destination the runs it does not hold yet, oldest first within each downpipe. Expect at least an hour (how long the wait takes).
  3. Try the removal. While runs are still held only on the old destination, the refusal says how many. The count falls as the copies land. Refusal two states when the count holds, and that case applies to this count too. With two or more owners, the refusal and its count go to the owner who approves, and each attempt is a new approval request that raises a destination-change alert.
  4. When the refusal no longer counts runs and says “Reassign them first.”, take the old destination off those downpipes.
  5. Remove the old destination. When no run is held only there, it needs no force.

After you remove the old destination, the remaining destination holds the recovery bundle that replication copied there once the downpipe had a run. If that copy has not landed yet, the engine retries it at the next cron tick. Until it lands, recover from the destination with --signer and the kit’s signer.pub (offline recovery).

While the old destination stays on a downpipe, a new run of that downpipe can seal to it. If that run is not copied before step 4, the removal in step 5 is refused. The refusal then says that the downpipe has fewer than two destinations. Tick both destinations on that downpipe again and repeat from step 2.

At each tick, the replication pass has its own time allowance and shares the tick’s subrequest budget. When either runs out, the pass stops, and the next tick continues from where it stopped (runReplications, engine/src/seal/replicate.ts). The number of runs one tick copies depends on the size of the runs and on how fast the two destinations answer.

A removal waits for a second owner’s approval when the account has two or more owners, or when dual control is on (four-eyes change control). The engine then applies both refusals when the second owner approves, so the approver gets the refusal and its count, and the request stays pending. The request returns no notice, because the removal has not run. When the approval runs the removal, the approve response carries the notice described below, only when there is one. The notice is a top-level field of the POST /admin/owner-actions/<id>/approve response (approveOwnerAction, engine/src/sched/scheduler-do-dual-control.ts). When the approve response carries a notice, the console shows it in a warn toast; with no notice, the console shows “Owner action approved” (approveButton, console/src/screens/owner-actions.ts).

How long the wait takes

The replication fairness test uses 5 downpipes with 3 runs each, so 15 runs to copy to the new destination. With the normal allowance per tick, the new destination holds all 15 runs within 4 ticks (about 1 hour). With about 20 subrequests left per tick, it holds them within 35 ticks (about 8 hours 45 minutes). In the slow case, no run is held while replication copies the segment data the runs share. From the first held run, one run is held at every tick (engine/test/validate-repl-fairness.ts).

The wait grows with the number of runs to copy and with the amount of data that is not yet on the new destination. Expect at least an hour, and the wait can take several hours or longer. The count can stay the same for some ticks before it falls.

Do not use force while the count is above zero, unless you accept that downpipes cannot restore those runs.

Refusal one: a downpipe pins it

If any downpipe names this destination explicitly, as its primary or as a replica, the removal is refused and there is no force for this one. The message names the downpipes.

The reason it is absolute is that a pinned destination id never silently falls back to something else. Removing it would leave those downpipes pointing at an id that does not exist. A downpipe with no other destination would fail its next run rather than quietly write somewhere else. A fan-out downpipe would skip the missing id and keep fewer copies than you configured.

What the refusal says next depends on whether this destination holds runs that no other destination holds yet. When it holds none, the refusal gives one step. With two downpipes named nightly-kv and orders-db, it reads:

destination is in use by 2 downpipe(s): nightly-kv, orders-db. Reassign them first.

When it holds such runs, the refusal tells you to keep the destination on the downpipes, and it counts the runs. With six runs held only here, it reads:

destination is in use by 2 downpipe(s): nightly-kv, orders-db. Keep it on them for now: it holds 6 backed-up run(s) that no other destination holds yet (nightly-kv, orders-db). Replication copies them to the other destinations of their downpipes at each 15-minute tick, oldest first within each downpipe. This can take hours. It stops copying for a downpipe left with one destination. When a removal no longer reports runs held only here, take this destination off those downpipes and remove it.

Do not take the destination off a downpipe while it holds that downpipe’s only copies. The downpipe then has one destination, replication does not copy its runs, and the second refusal below stops the removal.

Some of the counted runs can belong to a downpipe with fewer than two destinations. The refusal then states how many and names the step that makes replication run for them. With six runs, two of which belong to such downpipes, it reads:

destination is in use by 2 downpipe(s): nightly-kv, orders-db. Keep it on them for now: it holds 6 backed-up run(s) that no other destination holds yet (nightly-kv, orders-db). Replication copies 4 of them to the other destinations of their downpipes at each 15-minute tick, oldest first within each downpipe. This can take hours. It stops copying for a downpipe left with one destination. 2 of them belong to downpipes with fewer than two destinations, which replication does not copy, so waiting does not clear them. Give each of those downpipes this destination and one other, and wait for replication to copy the runs. This can take hours too. When a removal no longer reports runs held only here, take this destination off those downpipes and remove it.

A refusal names at most five downpipes, and adds , … when there are more.

Refusal two: it holds the only proven copy

This is the data-loss guard, and it is the one force overrides. It applies when no downpipe names this destination but some runs have their only proven copy here.

Under 3-2-1 failover a run may have sealed here and not yet been replicated anywhere else. This destination is that run’s recorded origin, and for the moment it is the only proven copy. A run also counts here when its recorded origin destination has since been removed and this destination is the one proven to hold it.

A run stops counting against this guard when its index is inside the recorded window of another destination. It also stops counting when that destination lists the run among the runs it holds above the window (holdsAbove). The window starts at that destination’s floor, the lowest run index it has contiguously observed (holdsFrom). It ends at the newest run it holds with no gap below it (holdsIndex). A destination with no recorded floor contributes no coverage at all, so the removal stays blocked. That is deliberately the safe direction: a replica that only started observing at the history ring’s floor cannot prove it holds a run from below that floor.

When replication cannot copy a run, for example because no destination holds it, it still copies that downpipe’s newer runs. It records them in holdsAbove, so they leave the count (processTarget, engine/src/seal/replicate.ts). Only the run it cannot copy stays in the count, until that run leaves the downpipe’s newest 50 runs.

Only a destination still configured proves either kind of coverage. A removed destination’s old replication record proves nothing.

A run that seals to a destination moves that destination’s holdsIndex up only when every earlier run above the old value also sealed there. Otherwise the replication pass moves holdsIndex up when it has copied the runs in between (gapBelowSeed, engine/src/sched/destinations.ts). A new run on the new destination therefore cannot make the guard count older runs as copied while they are only here.

The refusal names the count and the downpipes, and then the step that ends it. When the downpipes have two or more destinations, the step is a wait that ends:

destination is the only proven copy of 4 backed-up run(s) (nightly-kv). Replication copies them to another destination of their downpipes at each 15-minute tick, oldest first within each downpipe. This count falls as replication copies runs. It can stay the same for several ticks while replication copies shared data, and the wait can take hours. If replication cannot copy a run (for example, no destination holds it), that run stays in this count until it leaves the downpipe’s newest 50 runs. Or remove it with force: downpipes then cannot restore those 4 run(s).

When the downpipes have fewer than two destinations, waiting does not help, and the refusal names the step that makes replication run:

destination is the only proven copy of 6 backed-up run(s) (nightly-kv, orders-db). These runs belong to downpipes with fewer than two destinations, which replication does not copy, so waiting does not clear them. To keep them, give each of those downpipes this destination and one other, wait for replication to copy the runs (this can take hours), then take this destination off them and remove it. Or remove it with force: downpipes then cannot restore those 6 run(s).

When only some of the runs belong to such downpipes, the refusal carries both parts, each with its own count. With six runs, two of which belong to such downpipes, it reads:

destination is the only proven copy of 6 backed-up run(s) (nightly-kv, orders-db). Replication copies 4 of them to another destination of their downpipes at each 15-minute tick, oldest first within each downpipe. This count falls as replication copies runs. It can stay the same for several ticks while replication copies shared data, and the wait can take hours. If replication cannot copy a run (for example, no destination holds it), that run stays in this count until it leaves the downpipe’s newest 50 runs. 2 of them belong to downpipes with fewer than two destinations, which replication does not copy, so waiting does not clear them. To keep them, give each of those downpipes this destination and one other, wait for replication to copy the runs (this can take hours), then take this destination off them and remove it. Or remove it with force: downpipes then cannot restore those 6 run(s).

Unpinned downpipes count too. A downpipe with no explicit destination seals to the default, and its run records the default’s id, so removing that destination counts those runs against the guard. The guard protects you when you pin nothing.

A run that records no id counts against the current default and against the 32 most recent earlier defaults (couldHaveTakenDefaultRoutedRuns, engine/src/sched/scheduler-do-dest-config.ts). When the default moves, the engine writes the old default’s id onto those runs.

A downpipe that names no destination has fewer than two, so replication does not copy its runs. Give it this destination and one other before you wait.

Runs older than the run history

The guard and the replication pass read only the runs in each live downpipe’s run history, which keeps the newest 50 runs of each downpipe. Replication does not copy an older run, and waiting does not help it. The guard does not count the runs of a deleted downpipe.

When a downpipe with a run from this destination in its history also has older runs, each refusal ends with their count:

4 older run(s) of the downpipes that sealed runs here are no longer in the run history, which keeps the newest 50 runs of each downpipe. Replication does not copy those, so downpipes cannot restore any of them held only here once this destination is removed, with or without force.

That count is every run of those downpipes that has left the history. Some of those runs can have sealed to another destination, and the sentence says “any of them held only here” for that reason.

What forcing actually costs

Forcing does not merely un-reference those archives. The destination’s stored credential goes with it, so the engine can no longer address the bytes. Here is what that does and does not mean. Nothing in the bucket is deleted: removal filters the entry out of the stored destination collection and touches no object. What is destroyed is downpipes’ copy of the credential, and a restore of an affected run then fails with the engine’s own terminal reason, which names the remedy: this run’s only copy was on a removed destination; re-add it (or a destination that holds these bytes) and retry, or re-point the restore with an explicit destination (REASON_ORIGIN_REMOVED, engine/src/restore-reasons.ts).

So the recoverable case and the unrecoverable one turn on where the access key lives. If you still hold it in your own provider console, you can re-add a destination pointed at the same endpoint and bucket. The re-added destination gets a new id, so the restore must name it as an explicit destination. If the only copy of that key was the one downpipes held, the bytes are stranded for good.

So the guard is not being cautious about tidiness. It is the difference between N runs you can still restore and N runs you cannot.

Forcing is still allowed, because sometimes dropping those copies is the deliberate decision. When you do, the engine records how many runs it just orphaned, computed before the removal and kept as the magnitude of the loss. That count exists so the event is auditable later rather than being reconstructed from what someone remembers choosing.

A forced removal that drops runs returns a notice that states how many:

Destination removed with force. Downpipes cannot restore the 6 backed-up run(s) that were only here.

When the downpipes also have runs older than the run history, the notice adds their count:

4 older run(s) of the downpipes that sealed runs here are outside the run history, which keeps the newest 50 runs of each downpipe, and replication does not copy runs outside it. Downpipes cannot restore any of them that were only here.

A removal that drops no run but has older runs outside the history returns a notice that starts “Destination removed.” and continues with the same sentence. This applies with or without force. A removal that drops nothing the engine knows of returns no notice (destRemovedNotice, engine/src/sched/dest-removal-text.ts).

Usually the right move is to wait

For a downpipe with two or more destinations, this guard clears itself. Replication copies the runs, the count falls as the copies land, and then the removal succeeds with no force at all. For a downpipe with fewer than two destinations, waiting does not clear its runs. The refusal says which of the two applies.

Three options, in the order worth trying them:

  1. Wait, when the refusal says replication copies the runs. Try the removal again after a tick to read the new count. Expect at least an hour, and the count can stay the same for some ticks before it falls (how long the wait takes). With two or more owners, the owner who approves reads the count. The count is the measure to use here: the redundancy map’s Copies row shows only which destinations hold each downpipe’s latest successful run.
  2. Give each downpipe this destination and one other, when the refusal says waiting does not clear the runs. Then wait, and take this destination off the downpipes when the refusal says “Reassign them first.”
  3. Force, having decided those copies are expendable.

How the console shows a refusal

The console asks you to confirm the removal first. The console can then ask you to confirm with your own passkey. After a refusal, the console shows the engine’s own sentence:

  • For the first refusal, a message reads “Could not remove the destination.” and then the engine’s sentence. The console offers no force, because force does not clear this refusal.
  • For the second refusal, the console opens a dialogue titled “Force-remove this destination?”. It shows “The engine refused with:” and the engine’s sentence. To force the removal, you type the destination’s name and select Force-remove (offerForceRemove, console/src/screens/destination-cards-actions.ts).

The console shows the engine’s sentence in full, never cut short. For the second refusal, the force-remove dialogue shows the sentence in its own scrollable box. The type-to-confirm field stays in view. The full sentence is also in the error field of the POST /admin/destinations/remove response.

When a removal succeeds and the response carries a notice, the console shows the notice in a warn toast. When there is no notice, the console shows “Destination removed.” The notice is in the response of POST /admin/destinations/remove, or in the approve response when a second owner approves the removal.

The default is promoted silently

One side effect worth knowing before removing anything: if you remove the current default, the first remaining destination in the list silently becomes the default. It is the first one, not the one that sat next to the one you removed: the engine filters the removed entry out and takes position zero of what is left (removeDest, engine/src/sched/scheduler-do-dest-config.ts). Nothing asks you to confirm which one, and it is chosen by position rather than by any preference of yours.

So after removing a default, check which destination is now the default before the next run seals to it. Every unpinned downpipe follows that choice.

What is recorded

A removal the engine accepts, or queues for a second owner’s approval, raises a destination-change alert. The alert says explicitly when force was used. Your notifications can then tell a forced drop from an ordinary one, not only the engine’s own records.

The two refusals are recorded as separate closed classes, in-use and orphan-guard, so “a downpipe still names this destination” and “the engine refused to drop the only proven copy” are distinguishable after the fact. Both classes are engine refusals. An operator who reads a refusal and walks away leaves no record of its own: what is recorded is the refusal, not the decision that followed it. A forced removal, by contrast, is recorded, with the count of runs it orphaned and up to five of their downpipe names.

Where this fits

Last updated .