Reading the redundancy map: N of M copies, catching up, and down
The redundancy map shows how many proven copies a fan-out downpipe actually has, which destinations are unreachable, and which are still catching up. This page is for the operator reading that view, so you can tell a normal post-run lag apart from a real shortfall and trust the numbers as proven outcomes rather than guesses.
Everything on this view is derived from per-destination replication state the engine records. That state captures what each destination is proven to hold and whether the last attempt to it succeeded. It is never inferred from staleness, the canary, or any other signal. For the engine machinery underneath it, see failover and replication. For the whole-estate picture this drawer sits inside, see the topology map.
The Copies row, field by field
Open a fan-out downpipe in the map’s drawer and you get a Copies row. It reads as N of M copies, optionally followed by a state. M is the number of destinations this downpipe is configured to fan out to (its intended width). N is how many of those destinations are proven to hold the latest successful run.
| You see | It means |
|---|---|
3 of 3 copies | Every configured destination is proven to hold the latest successful run. Full redundancy. |
2 of 3 copies. Replication runs every 15 minutes and can take hours. | At least one reachable destination does not yet hold the latest run, and no destination is down or never reported. It is mirroring on its own schedule, and the data is safe on the copies that do hold it. |
2 of 3 copies - 1 never reported | One destination has produced no replication record at all, and at least two successful backups have completed since it joined the fan-out. This is not behind, it is proven to be making no copy, and waiting will not fix it on its own. |
2 of 3 copies - 1 down | One destination’s most recent seal or mirror attempt failed, so it is treated as unreachable. The label names the down count first. It names never-reported only when no destination is down, and it shows the 15-minute note only when neither applies. A down destination can add its cause in brackets, such as a rejected credential. |
3 destinations (no run yet) | A fan-out downpipe that has not yet had a successful run. There is nothing to replicate, so this is a pending state, not a shortfall. |
A destination counts toward N only when its proven holdsRunId equals the downpipe’s latest successful run id. Anything less, whether reachable-but-behind, never-reported, or unreachable, does not count, so the figure cannot read higher than the recorded replication state.
Never reported is not the same as catching up
A destination added minutes ago has no replication record yet, and reads as catching up: that is correct, ordinary behaviour, not a fault. The console only calls a destination never-reported once at least two successful backups have completed since it joined the fan-out and it still holds none of them. Below that bar it counts as catching up, as a newly added destination should. At or above it, the silence has outlasted backups that did complete, so the console stops advising you to wait and says so instead.
The Copies row is shown only for fan-out downpipes
A downpipe with a single destination has no redundancy concept to show, so the Copies row is suppressed for it. If you do not see a Copies row, the downpipe writes to one destination. Configure a second destination to get redundancy and the map reading for it. See multiple destinations.
The five redundancy states
The roll-up that backs the Copies row resolves to one of five states. Knowing which state you are in tells you whether to act.
| State | When | What it tells you |
|---|---|---|
single | One destination only | No redundancy concept. The map drawer suppresses the Copies row entirely. |
pending | Fan-out, but no successful run yet | Nothing has been captured to replicate. Not a shortfall; wait for the first good run. |
full | Proven copies equal the intended width | Every configured destination holds the latest run. The healthy steady state. |
partial | At least one proven copy, but fewer than intended | The data is safe on the copies that hold it. Some destinations are catching up, never reported, or down. |
none | Zero proven copies of the latest run | No destination is yet proven to hold the latest run. Worth investigating, especially alongside a down badge. |
A partial state right after a run is the expected case, not a fault. It resolves on its own once replication catches the remaining destinations up.
Per-destination edges and the down badge
The map draws one edge per destination a downpipe writes to, rather than a single shared edge. Each edge carries its own status, layered over the downpipe’s own capture status, so no single destination’s lane can read greener than the run itself.
For a downpipe whose capture is healthy, each destination lane reads as follows.
| Lane reads | Condition |
|---|---|
| Healthy (the downpipe’s own status) | The destination holds the latest successful run. |
| Amber, partial | The destination is reachable but does not yet hold the latest run. The data is safe elsewhere; this copy is catching up. |
| Amber, no copy | There is no restorable copy on this destination yet. Either the engine reaches it but it holds no run, or it has never reported holding a copy and at least two successful backups have completed since it joined the fan-out. Ranked worse than partial (a copy that is behind is being made; one that has stayed empty is not) and better than failed (the data is still safe on the destinations that do hold it). |
| Failed | The destination’s last attempt failed, so it is unreachable, regardless of which runs it once held. |
If the downpipe’s own capture is failed, disabled, or in an unknown state, every lane shows that capture state unchanged. The data itself is not safe or not measured, so the redundancy overlay never paints a lane greener than the run. A destination added moments ago, before it has a replication record at all, reads as partial rather than no copy: no-copy is reserved for a destination the evidence proves has stayed empty, never for one that simply has not reported yet.
A destination node that a fan-out downpipe writes to carries a down badge the moment any downpipe reports it unreachable. Because one destination is often shared by several downpipes, the badge is aggregated across them: one downpipe finding the destination unreachable is enough to badge it down for the whole map. A destination that only single-destination downpipes write to gets no down badge. The failed run of that downpipe shows the outage instead.
Why catching up right after a run is normal
This is the single most important expectation to set. A copy showing catching up immediately after a run is the normal, healthy case, not a problem.
A run is first sealed to one destination, and replication to the other destinations runs as a separate scheduled pass, not inline during the seal. The replication pass runs after the seal loop on each fifteen-minute tick. A run that finishes during the tick can reach the other destinations later in that tick. A run that finishes on the seal object’s alarms, or from Run now, waits for the next tick. In the window between the seal and that pass, the other destinations do not hold the new run yet, and the map shows that.
So a downpipe that just ran will often read 2 of 3 copies. Replication runs every 15 minutes and can take hours. until replication completes, then settle to 3 of 3 copies. A recovered destination is likewise re-probed on the next tick rather than the instant it comes back. Read the replication wait here as on a tick, not in real time.
When catching up is worth a look
A copy that stays in catching up across many ticks, or a none reading that does not resolve, is worth investigating, usually together with a down badge or a destination whose credential or bucket changed. A single tick of catching up right after a run is expected and needs no action.
Why the numbers cannot over-claim
The view rests on one rule: a destination’s recorded holdsIndex advances only to the highest contiguous run it holds, with no gap below it. The first missing run stops it.
That matters because runs can land on a destination out of global order under failover and backlog catch-up. If a destination holds runs 1, 2, and 4 but is missing run 3, the recorded index stays at run 2, not run 4. Recording a higher index would over-claim a copy the destination lacks, and inflate the Copies count. A run that no destination can supply freezes the recorded index below it until a later pass copies it. A destination still proves the newer runs it holds above that frozen window: the replication pass records them in a separate list, and the removal guard reads the window and the list together, so a stranded run does not hide a copy of a later run from the guard (engine/src/sched/scheduler-do-dest-config.ts).
When a run’s origin destination no longer exists, replication copies it from another destination that already holds it. Only a run no destination holds stays stranded, and it holds the index until it leaves the run history.
A seal follows the same rule. A run that seals to a destination raises its recorded index only when every earlier run above the old index also sealed there (gapBelowSeed, engine/src/sched/destinations.ts). A destination you add to a downpipe as its primary therefore reads as catching up after it takes the latest run. It stays catching up until replication has copied to it the older runs in the run history.
The recorded state also moves forward only by index, so a late or duplicate record can never rewind a destination’s proven copy. And a reachable destination that could mirror nothing this pass refreshes its reachability heartbeat without advancing what it holds, so an outage and a quiet-but-current copy are never confused.
The fields behind the row
The console reads each destination’s replication state, which carries the latest run it is proven to hold and whether the last attempt to it succeeded.
| Field | Meaning |
|---|---|
holdsRunId and holdsIndex | The latest run the destination is proven to hold, by id and by account-global index. Advances only forward by index, and only to the highest contiguous run with no gap below it. |
lastOk | Whether the most recent seal or mirror attempt to this destination succeeded. This is the heartbeat behind the down badge. |
lastAttemptAt | The time of that most recent attempt. |
engineVersion | The engine version that wrote this state. A record written without a version has none. In the support pack, a row whose version is not the running one, or is absent, carries stale: true. |
copy | In the support pack, none-yet on a row whose last attempt succeeded while the destination holds no run. There is no restorable copy there yet, whatever lastOk says. |
reason | A coarse, secret-free reason when the last attempt failed. A failed mirror records one of ten: unreachable, not-configured, runlog-unreadable, runlog-corrupt, copy-failed, integrity-verify-failed, origin-runlog-unreadable, origin-run-missing, origin-dest-unconfigured, deploy-binding-missing (REPLICA_FAULT_REASONS, engine/src/seal/seal-faults.ts). deploy-binding-missing means that the destination is the deploy-time destination and the deployment declares no environment destination behind it. A failed seal records one of eight: auth, worm-refused, throttled, timeout, network, tls, unreachable, other (REPLICATION_REASONS, engine/src/sched/sched-fault-core.ts). |
The roll-up compares each destination’s holdsRunId against the downpipe’s latest successful run id to compute N, collects every destination whose lastOk is false into the down set, and treats every reachable-but-behind destination as catching up. None of these inputs is a staleness estimate or a canary reading; each is a recorded outcome of a real seal or mirror attempt. This data carries no key material and no plaintext.
The map reading is not the posture control
It is worth keeping two different things apart. The redundancy map shows which destinations the recorded replication state lists as holding the latest run. The redundant-copies posture control reads the same records, but grades on its own schedule. For each downpipe, it grades every non-origin destination on the newest successful run that the replication pass has visited it for. After six hours with no visit, it grades on the newest run outright. It counts a destination only when the destination holds that run.
So the two can disagree for a short time. Right after a backup, the map can read partial while the posture control still passes on the run before. A destination that holds no run never counts toward either. You should not read one as proving the other: the map is the live view, and the posture control is the graded view.
Where this fits
Failover and replication
The engine passes underneath this view: seal to the first reachable destination, then replicate each destination’s backlog on a later tick.
The topology map
The whole-estate map this drawer opens from, with sources, destinations, and copy coverage across every downpipe.
3-2-1 redundancy
What three copies on two media with one off-site means here, and what is verified and what is attested.
Multiple destinations
Configuring more than one destination per downpipe so there is a redundancy map to read in the first place.
Last updated .