How seal-to-first-reachable failover and replicate-after-seal work

This page explains how downpipes keeps a backup running when one destination is down, and how it then mirrors each run to the other destinations. It is for the developer or operator who wants the mechanism, including where its bounds are. The two mechanisms are separate: failover decides where a single run is captured right now, and replication catches every destination up afterwards on its own schedule.

For the broader 3-2-1 picture and how to configure several destinations, see 3-2-1 redundancy and multiple destinations. This page is the how-it-works for the two engine passes underneath them.

Nothing here is instant. Replication is bounded by the engine’s roughly every-fifteen-minutes scheduled tick. Failover probes at the start of each run, scheduled or manual, so a recovered destination is re-probed on the next run. Read every “catches up” below as “on a tick”, not “in real time”.

Failover: seal to the first reachable destination

Before each seal, a downpipe that fans out to two or more destinations probes each one in the configured order (the primary first, then each replica). It seals to the first reachable one. So a down primary falls over to a healthy replica, and one bucket being down does not lose the run. When every destination is unreachable, the run does not silently disappear: the engine records a failed run whose error is “all destinations unreachable” and frees the lease so the next tick retries, and the per-destination down state is visible in its own right.

A single-destination downpipe fails loud: it runs no probe and no failover, and a pinned-but-unconfigured destination id throws rather than silently writing to the environment fallback, which would split the archive and break its chain. Failover exists only where there is a sibling destination to fall over to.

The deploy-time destination deploy takes part in failover and replication like any other destination. When the deployment declares no environment destination, each path refuses deploy with the reason deploy-binding-missing.

The reachability probe writes, it does not just list

The probe is the gate that decides “reachable”, and it tests writability rather than mere reachability, because a read-only probe would pass on a credential that can list but not write, and the seal would then fail with no second chance to fail over. The probe writes a one-byte marker and deletes it, using the same kind of PUT the seal needs.

Because the probe writes with the same operation the seal uses, every cause of “down” is caught at selection time. The run therefore fails over to a healthy destination instead of failing the seal.

Origin recording: which destination a run actually sealed to

Because failover can make a run seal somewhere other than its configured primary, the engine records the destination each run actually sealed to. The engine writes that origin onto the successful run’s history row. For a downpipe that names a destination, the seal also records a heartbeat for that destination. It raises the destination’s proven index only when no earlier run is missing there (see below). A failed run sealed nowhere, so it records no origin, and a later resolver falls back to the configured primary.

This recorded origin is load-bearing for the rest of the system: the replication pass mirrors a run from its origin first, and a restore reads a run from its origin first. It is the difference between “where this run was supposed to go” and “where this run actually is”.

Replication is a separate cron pass, not a fan-out during the seal

Replication does not happen inline at seal time. It runs as its own scheduled pass (runReplications), and that is deliberate: a large run finalises later on the per-downpipe seal Durable Object’s alarms, so the bytes to mirror are not all present at the moment the seal is first triggered. The pass re-checks every tick and is fail-open per downpipe and per destination, so a fault on one is logged and skipped without ever failing a sealed run, whose origin copy is already safe, and the next tick re-attempts.

Think of the seal as making one good copy, and replication as the background worker that fans that copy out to the other destinations afterwards.

Catch-up scope: the whole backlog, oldest first

The pass does not mirror only the latest run. For each destination it catches up on its whole backlog, every run it is missing, oldest first. It copies each missing run from that run’s own recorded origin. When the origin cannot supply the run, it copies from another destination of the same downpipe that already holds the run. Mirroring out of global order is safe because each destination gets its own re-linked, index-ordered chain (covered below), so a back-filled older run never forges a rollback.

This is also how a destination you add to a downpipe gets the older runs. The next ticks copy to it every successful run still in the downpipe’s run history, oldest first. The run history keeps the newest 50 runs of each downpipe, and the pass does not copy a run older than that. The pass runs only for a downpipe with two or more destinations, so keep the old destination on the downpipe until the new one holds its runs. Remove a destination gives the order.

Each tick bounds the work. The pass has its own time allowance, and it shares the tick’s subrequest budget with the passes before it (passWindow, engine/src/seal/budget.ts). When either runs out, the pass stops. A run it did not finish gets no done-marker, and the next tick copies only the objects that are still missing.

The pass is idempotent. A destination’s own RUNLOG already carrying a run id is the done-marker, written last after the objects land, so a run that is already present is skipped cheaply with a single RUNLOG read. That ordering, objects first and the RUNLOG entry last, is what makes a present run cheap to skip and an interrupted mirror safe to retry.

Per-replica integrity: a keyless verify before the done-marker

A mirror is not trusted just because the copy loop finished. Before the run’s done-marker is written to a replica’s RUNLOG, the engine runs a keyless post-copy integrity check on that replica: it confirms the signed root verifies and that every shard is present and hashes to the signed root. The check also confirms that the pending RUNLOG entry agrees with that root on the run, the downpipe and the RUNLOG index. A silent corruption, a truncated copy during the get-and-put, or a disagreeing entry therefore throws, and the replica is never recorded as holding a good copy; the target is recorded down and retried on the next tick.

The anti-rollback flag is intentionally not required for this check, because the replica-local RUNLOG entry that proves freshness is appended immediately afterwards, so only the signature and completeness apply at this point. Each replica then gets the run appended to its own RUNLOG with a re-linked, replica-local prevRunId, so a destination added mid-life, or one that skipped a run under failover, keeps a valid, fork-free per-downpipe chain rather than a dangling cross-bucket pointer that a reader would read as a rollback.

That append reads the replica’s stored RUNLOG, verifies it under the engine’s own key, and re-signs it the same way a seal does. It refuses to write when the check fails, and a RUNLOG that does not verify waits for an Owner’s attended re-sign. Restrict write access on every destination to the engine’s own credentials. Use a retention lock where the destination supports it.

“N of M copies” comes from proven outcomes

The per-destination state the console reads to show “N of M copies” and the per-destination down indicator is recorded from proven outcomes, not inferred. Each destination’s row holds the latest run it is proven to hold and whether the most recent seal-or-mirror attempt to it succeeded.

The index a destination records is the highest contiguous run it holds with no gap below it. It is not merely the latest run that happened to arrive. Recording a higher index would over-claim a copy the destination lacks and poison both the copy count and the removal orphan-guard, so a run that no destination can supply freezes the recorded index below it until a later pass copies it.

When a run’s origin destination no longer exists, replication copies the run from another destination of the same downpipe that already holds it. Only a run no destination holds stays stranded, and it holds the index until it leaves the run history, which keeps the newest 50 runs.

A seal follows the same rule. A run that seals to a destination raises its recorded index only when every earlier run above the old index also sealed there. Otherwise the index stays, and the replication pass raises it when it has copied the runs in between (gapBelowSeed, engine/src/sched/destinations.ts).

Bounded gaps

Two bounds follow from running on a periodic tick with a bounded history.

First, a replica trails its origin. Replication runs on the same roughly every-fifteen-minutes schedule, so a freshly sealed run reaches the other destinations on a later pass, not the instant it is sealed. A pass that runs out of its time allowance or of the tick’s subrequest budget resumes on the next tick. The pass also tries a recovered destination again on the next tick, not immediately.

Second, the pass copies only the runs in the run history, which keeps the newest 50 runs of each downpipe. It never copies an older run: not to a destination that was offline while that run left the history, and not to a destination you add later. Copying those would need a multi-source RUNLOG scan the pass does not do. When you remove a destination, the refusals and the removal response state how many such older runs its downpipes have (engine/src/sched/dest-removal-text.ts).

The disaster-recovery payoff (a forward pointer)

The reason all of this matters shows up at recovery time. Each destination is an independent, restorable mirror with its own valid chain. That is why a restore, a drill or a verify falls back transparently to a replica holding the run when the primary bucket is lost. The fallback retries another destination for any availability fault: a run-tree object is missing (a missing segment counts as an integrity failure), the destination refused or faulted with an HTTP status (a revoked credential, a provider outage, a redirect, a rate limit), or a network or transport fault reading it. An integrity failure returns immediately rather than masking a real problem, because each restore-class operation opens the run before any write, so a fallback never half-applies.

A refused operator input also returns immediately and is not a finding: a pinned signer fingerprint that is incomplete, or a destination with no bundled signer.pub to check it against. A replica holds the same bytes and the same pin, so it could only refuse again. The reason reads “restore input cannot be used”, and it does not call the archive corrupt (REASON_USAGE, engine/src/restore-reasons.ts; usageError, engine/src/format/integrity-error.ts). The restore experience itself, including how recovery point is expressed, lives in the Recovery section.

A freshness fault splits two ways, and only one of them is terminal

The anti-rollback check is the one place where “a replica holds the same signed run” stops being true, so the engine treats its two outcomes differently.

The second row is the case three copies exist for: one copy is damaged and the others may be perfect. Nothing is weakened to get it. The engine opens the replica and verifies it in full from its own bytes. The operation re-runs rather than inheriting anything from the failed attempt. A run served from anywhere other than the first destination tried says so in its result and inside its signed receipt core.

Where this fits

Last updated 25 October 2026.