How seal-to-first-reachable failover and replicate-after-seal work

This page explains how downpipes keeps a backup running when one destination is down, and how it then mirrors each run to the other destinations afterwards. It is for the developer or operator who wants the real mechanism rather than a slogan, including where the bounds are. The two mechanisms are separate: failover decides where a single run is captured right now, and replication catches every destination up afterwards on its own schedule.

For the broader 3-2-1 picture and how to configure several destinations, see 3-2-1 redundancy and multiple destinations. This page is the how-it-works for the two engine passes underneath them.

Nothing here is instant. Both passes are bounded by the engine’s roughly every-fifteen-minutes scheduled tick, and a recovered destination is re-probed on the next tick. Read every “fails over” and “catches up” below as “on a tick”, not “in real time”.

Failover: seal to the first reachable destination

Before each seal, a downpipe that fans out to two or more destinations probes each one in the configured order (the primary first, then each replica) and seals to the first reachable one. So a down primary falls over to a healthy replica, and one bucket being down no longer means the run is lost. When every destination is unreachable, the run does not silently disappear: the engine records an honest failed run whose error is “all destinations unreachable” and frees the lease so the next tick retries, and the per-destination down state is visible in its own right.

A single-destination downpipe keeps the prior fail-loud path verbatim: no probe, no failover, and a pinned-but-unconfigured destination id throws rather than silently writing to the environment fallback, which would split the archive and break its chain. Failover exists only where there is a sibling destination to fall over to.

The reachability probe writes, it does not just list

The probe is the gate that decides “reachable”, and it tests writability rather than mere reachability, because a read-only probe would pass on a credential that can list but not write, and the seal would then fail with no second chance to fail over. The probe writes a one-byte marker and deletes it, using the same kind of PUT the seal needs.

Because the probe writes with the same operation the seal uses, every genuine cause of “down” is caught at selection time and the run fails over to a healthy destination instead of failing the seal.

Origin recording: which destination a run actually sealed to

Because failover can make a run seal somewhere other than its configured primary, the engine records the destination each run actually sealed to. That origin is written onto the successful run’s history row and seeds that destination’s proven replication state, so the destination is now known to hold the run. A failed run sealed nowhere, so it records no origin, and a later resolver falls back to the configured primary, exactly as a single-destination deployment always did.

This recorded origin is load-bearing for the rest of the system: the replication pass mirrors a run from its origin, and a restore reads a run from its origin first. It is the difference between “where this run was supposed to go” and “where this run actually is”.

Replication is a separate cron pass, not a fan-out during the seal

Replication does not happen inline at seal time. It runs as its own scheduled pass (runReplications), and that is deliberate: a large run finalises later on the per-downpipe seal Durable Object’s alarms, so the bytes to mirror are not all present at the moment the seal is first triggered. The pass re-checks every tick and is fail-open per downpipe and per destination, so a fault on one is logged and skipped without ever failing a sealed run, whose origin copy is already safe, and the next tick re-attempts.

Think of the seal as making one good copy, and replication as the background worker that fans that copy out to the other destinations afterwards.

Catch-up scope: the whole backlog, oldest first

The pass does not mirror only the latest run. For each destination it catches up on its whole backlog, every run it is missing, oldest first, copying each missing run from that run’s own recorded origin. Mirroring out of global order is safe because each destination gets its own re-linked, index-ordered chain (covered below), so a back-filled older run never forges a rollback.

The pass is idempotent. A destination’s own RUNLOG already carrying a run id is the done-marker, written last after the objects land, so a run that is already present is skipped cheaply with a single RUNLOG read. That ordering, objects first and the RUNLOG entry last, is what makes a present run cheap to skip and an interrupted mirror safe to retry.

Per-replica integrity: a keyless verify before the done-marker

A mirror is not trusted just because the copy loop finished. Before the run’s done-marker is written to a replica’s RUNLOG, the engine runs a keyless post-copy integrity check on that replica: it confirms the signed root verifies and that every shard is present and hashes to the signed root. A silent corruption or a truncated copy during the get-and-put therefore throws, and the replica is never recorded as holding a good copy; the target is recorded down and retried on the next tick.

The anti-rollback flag is intentionally not required for this check, because the replica-local RUNLOG entry that proves freshness is appended immediately afterwards, so only the signature and completeness apply at this point. Each replica then gets the run appended to its own RUNLOG with a re-linked, replica-local prevRunId, so a destination added mid-life, or one that skipped a run under failover, keeps a valid, fork-free per-downpipe chain rather than a dangling cross-bucket pointer that a reader would read as a rollback.

“N of M copies” comes from proven outcomes

The per-destination state the console reads to show “N of M copies” and the per-destination down indicator is recorded from proven outcomes, not inferred. Each destination’s row holds the latest run it is proven to hold and whether the most recent seal-or-mirror attempt to it succeeded.

The index a destination records is the highest contiguous run it holds with no gap below it, not merely the latest run that happened to arrive. Recording a higher index would over-claim a copy the destination lacks and poison both the copy count and the removal orphan-guard, so a run skipped because its origin was unreadable freezes the recorded index below it until the gap is filled.

The bounded gaps, plainly

Two bounds are worth stating directly so there is no surprise. Neither is a regression; both follow from running on a periodic tick with a bounded history.

First, a replica trails by up to one tick. Replication runs on the same roughly every-fifteen-minutes schedule, so a freshly sealed run reaches the other destinations on a later tick, not the instant it is sealed, and a recovered destination is re-probed on the next tick rather than immediately.

Second, a destination offline longer than the run-history ring window will not have its aged-out runs back-filled. The backlog the pass works from is the runs the scheduler still retains in its history ring. A destination down for longer than that window loses the runs that aged out of the ring before it recovered, because recovering them would need a multi-source RUNLOG scan the pass does not do. In practice the ring spans many cadence periods and retention prunes old runs anyway, so this is a far smaller gap than the latest-only behaviour it replaces.

The disaster-recovery payoff (a forward pointer)

The reason all of this matters shows up at recovery time. Because each destination is an independent, restorable mirror with its own valid chain, a restore, a drill or a verify transparently falls back to a replica that holds the run when the primary bucket is lost. The fallback retries another destination for any availability fault: the object is missing, the destination refused or faulted with an HTTP status (a revoked credential, a provider outage, a redirect, a rate limit), or a network or transport fault reading it. An integrity failure returns immediately rather than masking a real problem, because each restore-class operation opens the run before any write, so a fallback never half-applies. The restore experience itself, including how recovery point is expressed, lives in the Recovery section.

A freshness fault splits two ways, and only one of them is terminal

The anti-rollback check is the one place where “a replica holds the same signed run” stops being true, so the engine treats its two outcomes differently.

The second row is the case three copies exist for: one copy is damaged and the others may be perfect. Nothing is weakened to get it. The replica is opened and verified in full from its own bytes, the operation re-runs rather than inheriting anything from the failed attempt, and a run served from anywhere other than the first destination tried says so in its result and inside its signed receipt core.

Where this fits

Last updated 5 August 2026.