The canary: known-answer backup assurance and the coalmine signal
The canary is a small synthetic backup of data the engine already knows. On a fixed cadence it seals that known data into an isolated corner of each destination, reads it back, decrypts it, restores it over an isolated cell, and compares every byte against the data it sent. While every byte returns exactly, the bird sings and the destination is sound to back up to. The moment one byte strays, the canary is dead and you treat that destination as suspect until it is investigated.
The metaphor is the old coalmine canary. Miners carried a bird underground because it would fall first if the air went bad, giving them warning while there was still time to leave. The downpipes canary plays the same role for your archive path: it is the thing that fails first, on data whose loss does not matter, so a destination that has quietly gone bad raises an alert before you ever need a real restore. This page explains the concept, the precise claim and its limit, the five liveness states, the multi-destination model, and how to read the canary screen day to day.
What the canary is
The canary is an account-level synthetic backup that is on by default. Every flight, the engine takes a small fixed corpus of known data that is generated in code from constant seeds, so the bytes are identical on every flight and on every deployment. That determinism is the point: the known answer is exactly known, so any deviation of even one bit is provable. The corpus is seven records spanning the byte shapes that catch real corruption, including every byte value from 0x00 to 0xFF, a run of all-zeros, a run of all-ones, multibyte text, structured JSON, and a kilobyte of deterministic pseudo-random bytes where a truncation or transposition anywhere shows up as a byte delta.
A flight exercises the real archive path end to end, only namespaced. The engine writes a probe, seals the known corpus through the identical seal pipeline a real backup uses, reads the archive back and verifies its signatures and its place in the RUNLOG chain, decrypts every record and compares it to the known data byte for byte, restores the recovered data over an isolated cell, and reads that cell back to byte-compare it again. Each object the canary writes lives under a dedicated _CANARY/runs/<runId>/ prefix inside whichever destination bucket it flies to, so it can never collide with your real archive, which lives under different prefixes. While every byte returns exactly, the canary is alive. The instant one byte strays, it is dead.
Known-answer means synthetic, by design
The canary never reads a single byte of your real data. It seals data the engine itself generated, into its own isolated namespace, and restores only into that namespace. That isolation is what lets it run continuously without ever touching a customer archive or a real binding.
The proof a green flight assembles
One flight runs a closed set of checks in order, and each is a distinct thing a green canary proves about that destination and the archive path.
| Check | What a pass proves |
|---|---|
| Can write | The destination is reachable, the credentials are valid, and write permission is confirmed by a real PUT |
| Can delete | Delete permission is confirmed against prior canary objects; an immutable bucket is noted, never failed |
| Sealed | The known corpus sealed to the destination as a real archive write, producing manifests and a RUNLOG entry |
| Signatures verify | On read-back, the root and shard manifest signatures verify, so the tamper-evidence holds |
| RUNLOG fresh | The RUNLOG entry is present, chained in linear order, and recent |
| Bytes intact | Every record decrypts, its per-record SHA-384 re-verifies, and the bytes match the known corpus exactly |
| Restored | The real restore path ran into the isolated canary cell, never a real binding |
| Restore verified | The restored cell read back and byte-compared exactly to the known corpus |
The byte-exact comparison against the known corpus is the load-bearing one. It is matched by exact bytes rather than by name, so it is order-independent, and a missing known record or an unexpected stray both count as a drift. The dead reason then reports precisely how far the data drifted, as a byte count.
What a green canary proves, and what it does not
This is the part to read slowly, because the easy mistake is to over-read the green.
A green flight proves the known-answer path for that destination at that hour. It proves the engine could write to the destination, seal a real archive, read it back with its signatures and RUNLOG intact, decrypt it, restore it, and recover every known byte exactly. It proves all of this without touching any of your real data.
What it does not prove is that any specific real backup is restorable. The canary exercises the same code paths your real backups use, which is strong evidence that the path itself works, but it carries known synthetic data, not your data. A green canary is a healthy path, not a per-backup recoverability certificate. For that, the engine has a separate keyed drill that decrypts and verifies your real in-scope records, and a keyless attestation that checks a real archive without a decryption key.
Evacuate the coalmine is a trust signal, not a data-loss claim
A dead canary means stop trusting that destination or path for real restores until it is investigated. It does not by itself mean any real data has been lost. The canary detected drift on its own known-answer corpus; your real archives are immutable and were not touched by the flight. The right reading is “do not rely on this destination yet”, not “the data is gone”.
The five liveness states
The canary reports one of five states, conveyed by colour and shape and a text label together, never by colour alone. Two of them are load-bearing. Alive means every byte returned exactly. Dead means a byte strayed, the data-integrity death that triggers the alert. The other three describe situations where nothing was proven either way.
The state that is most often misread is ailing. Ailing is a fault that stopped the check completing, for example the destination refused the credential, throttled the write, or answered with the wrong bytes. It is honestly distinct from a death: it is deliberately quiet and raises no evacuation alert, because a check that could not run is not detected corruption. Conflating ailing with dead would cry wolf, so the engine keeps them separate.
| State | Colour and shape | What it means | Alerts? |
|---|---|---|---|
| Alive | Green circle | The last flight returned every byte exactly; the full path is sound for this destination | No, it is the healthy state |
| Dead | Red square | A byte strayed on the last flight; do not trust this destination for real restores until investigated | Yes, a critical alert on the transition into dead |
| Ailing | Amber triangle | The flight could not finish, so nothing was proven; a check problem, not detected corruption | No, deliberately quiet |
| Pending | Neutral | The canary is on but has not completed a flight yet, or has no destination configured yet | No |
| Disabled | Neutral | The operator has turned the canary off; the path is no longer being checked between real runs | No |
The alert fires on the change, not every hour
A dead-to-alive recovery and an alive-to-dead death are notified only on the transition. A destination that stays dead does not page you every flight, and a recovery sends a single quieter notice. This keeps a persistent fault from becoming hourly noise.
The multi-destination model
The canary is not tied to one destination. As built, a flight flies the whole check set against every destination it is pointed at, so it proves each destination you back up to, independently. Each destination keeps its own result and its own liveness.
By default the canary flies to every configured destination, automatically including any you add later. The owner can instead pin an explicit subset on the canary screen. Whichever destinations it flies to, the headline status is the worst across them: the canary’s aggregate is dead if any one destination’s canary died, ailing if any could not complete and none died, pending if any has not yet flown, and alive only when every destination flown is alive. So a single failing destination turns the headline dead while the others may still read alive in their own rows.
There is a built-in cap on how many destinations one flight covers
To keep one flight inside the worker invocation budget, a single flight covers only the first twelve destinations. The rest are never flown, on that flight or any later one, because the truncation runs over the same ordered list every time. The excluded destinations do not appear on the canary screen either: the engine records the loss with their labels, and that record rides in a support bundle rather than in the console view. One downpipe can fan out to up to twenty destinations, so an account that fans out past twelve should read the canary as covering twelve of them and pin the canary to the destinations it most needs proven.
Cadence: roughly hourly, not on the dot
The canary is on by default and its configured interval is sixty minutes. It does not fire on the exact minute, though. It rides the engine’s existing fifteen-minute cron rather than its own trigger, so a due flight lands within fifteen minutes of its mark. The honest way to describe the cadence is on by default and roughly hourly within a fifteen-minute window. It is an assurance probe, not a stopwatch, and that window is the same effective floor the rest of the schedule works to.
How to read a dead canary
When a destination dies, the screen and the alert name the aspect that failed and report a byte count, and nothing more. The detail line is redaction-safe by construction: it carries the coarse fact only, never plaintext, never a key, never an object key, and never a real per-record hash. A typical dead reason reads as a named failed aspect followed by how many bytes strayed from the known corpus, for example that the integrity check found a number of records diverged and a number of bytes strayed.
The first move on a dead canary is to read which destinations failed, because the aggregate is dead if any one did. The screen lists each destination with its own liveness, and expands the failed ones so you see the aspect that died. Because a death is on known synthetic data, the investigation is about the destination and the path, not about recovering lost data.
Read the headline and the named destinations
The headline status is the worst across destinations. On a death, the screen names which destinations failed and shows the dead reason for the first one, so you know whether it is one destination or several.
Open the failed destination's checks
Each destination has its own row with the check set from its latest flight. The failed aspect carries the redaction-safe detail, naming what drifted and the byte count, so you can tell a signature failure from a byte-integrity failure from a restore failure.
Treat the destination as suspect, not your data as lost
A dead canary blocked and altered nothing. Investigate the destination or the path, and rely on a different destination meanwhile. Your scheduled backups kept running unchanged.
Safety, in one line each
The canary holds to three safety properties that follow directly from how it is built.
It is fail-open. A dead canary blocks nothing, alters nothing and deletes nothing of yours. It only alerts. A flight touches only the isolated _CANARY/ namespace and never a real backup or binding, so a canary fault can never affect a real run.
It is no-custody. All comparison happens engine-side in your own account. The console and the vendor see only liveness and counts, never a value, a key or a real hash. The byte-for-byte match is done by the engine; what leaves it is the verdict, not the data.
It is isolated. Every write and every restore lands under the dedicated _CANARY/runs/<runId>/ prefix, separate from your real archive. The restore-over step writes the recovered known data back into that same isolated cell, never a real or production binding, and the prior flight’s cell is cleaned up on the next flight so the residue stays tiny.
Where the canary lives
The canary has its own dedicated screen under Govern, which answers one question: is the canary singing, and what did its last flight prove. The screen shows the bird in its current state pose, the live status by colour and shape and label, the per-destination results with their check sets, a recent-flights history, and the owner controls to turn it on or off, choose which destinations it flies to, and fly it now.
There is no Overview status tile for the canary. The dedicated screen is the canonical place to read it. The screen also offers a “Preview a death” button, which is a pure client-side simulation: it lets an operator watch the bird fly and then die on fabricated data, without waiting for or causing a real death. It never calls the engine and never affects the live canary.
Deeper detail: isolation, determinism, the corpus and the no-false-death rules
The isolation wrapper. A flight addresses objects by their ordinary logical keys, and a thin wrapper prepends the _CANARY/runs/<runId>/ prefix to every operation. The wrapper adds no behaviour, it is a pure key rewrite, so the canary exercises the identical write, read, conditional-put, delete and list code paths a real backup uses, only namespaced. That is what makes the proof meaningful: it is the real path, not a stand-in.
The deterministic corpus. The seven known records are frozen in code and built from constant seeds with no clock and no randomness, so the bytes are identical on every flight and every deployment. The kilobyte block is filled by a tiny deterministic generator that is explicitly not cryptographic; it exists only to make a silent truncation, padding or transposition surface as a byte delta rather than hide in a block of zeros.
Why some flights are ailing, not dead. A configured destination that cannot be read yet reports pending rather than crying wolf before setup. A write probe that fails halts the flight as ailing, because an unreachable path is not proven drifted. A delete that an immutable bucket refuses is recorded as a note, never a failure, because lacking delete on an immutable destination is a posture observation, not a dead canary. The read-back runs in every key posture, so a break-glass-only engine reports a clean flight as alive rather than as a posture-driven ailing. A signature, freshness or integrity failure on the canary’s own freshly written archive, by contrast, is a real drift and is reported as dead.
Related reading
For the keyed drill that proves a real backup is recoverable, which the canary deliberately does not do, read prove recoverability. For the keyless check of a real archive without a decryption key, read immutability and attestation.
To understand the integrity chain the canary reads back, including the signatures and the RUNLOG, read verify at seal. For how proven-recoverability feeds the wider posture, read coverage and gaps.
For the no-custody trust model that the canary’s engine-side comparison rests on, read the no-custody trust model. For the honest phrasings the whole site is held to, including why this is described as tamper-evident and why a green canary is never over-claimed, read precise claims and honesty.
Last updated .