Skip to content
downpipes docs

What one canary flight checks: the eight aspects

Each canary run checks eight deterministic aspects, in order: write-probe, delete-probe, seal, read-signature, runlog-freshness, decrypt-integrity, restore, and restore-verify. A flight stops at the first death, so which aspect died is the most specific true statement about where a destination’s path broke. The canary itself, the coalmine metaphor, the five liveness states and how to read the screen are covered on the canary; this page assumes you have read that and goes one level down, into what a pass of each aspect proves.

It is written for a self-hoster or an auditor who wants to know precisely what evidence a green flight assembles, why the known-answer comparison is byte-exact, and why some flights honestly report ailing rather than a false death. Every fact here is taken from the canary code in the engine.

The eight aspects, in order

A flight runs a closed, fixed set of eight aspects against one destination, always in the same order, because each later aspect depends on the earlier ones succeeding. The set is the CanaryAspectKey union in types.ts, and the order is the order they are attempted in cycle.ts. A passing aspect is proven; a failing aspect is a death (a bit strayed where it must not); an aspect that could not be attempted is skipped; and a note is an informational observation that is not a death.

# Aspect What a pass proves
1 write-probe The destination is reachable, the credentials are valid, and write permission holds. A probe object is written and read back byte-exact through the real destination.
2 delete-probe Delete permission holds against the probe object. A refusal is a note, never a death, because an immutable bucket legitimately keeps the prune out.
3 seal The known corpus sealed to the destination as a real archive write through the identical seal pipeline, producing the manifests and a RUNLOG entry.
4 read-signature On read-back, the root and shard manifest signatures verify, so the tamper-evidence holds on the archive the flight just wrote.
5 runlog-freshness The RUNLOG entry is present, chained, and reads as the latest run in its namespace.
6 decrypt-integrity Every record decrypts, re-passes its own plaintext hash, and matches the known corpus byte-for-byte.
7 restore The real restore path writes the recovered known data back into the isolated canary cell, proving restore-side write permission.
8 restore-verify The restored cell is read back and byte-compared to the known corpus, so the round trip is whole.

The order matters for reading a result. A flight stops at the first death, so a death on aspect 6 tells you aspects 1 to 5 passed: the destination took the write, sealed a real archive, and read it back with its signatures and RUNLOG intact, but the decrypted bytes did not match the known data. The aspect that died is the most specific true statement about where the path broke.

Why each aspect is separate

The aspects are not redundant; each one isolates a distinct failure. The write and delete probes test the destination’s permissions before any real archive exists, so a credential or a bucket-policy problem surfaces as a probe result rather than a confusing seal error. The seal exercises the real write pipeline. The read-signature and runlog-freshness aspects prove the archive verifies and sits correctly in its chain. The decrypt-integrity aspect is the byte-exact known-answer compare, the load-bearing one. The two restore aspects prove the recovery write path and then re-confirm the bytes after a round trip through restore.

The What the last flight proved panel on the console's Canary screen. A destination named Default destination is badged Alive, with the age of its last flight beside it. Beneath it the eight aspects are listed in order, each with a Pass badge and a plain result: Can write, the destination accepted and returned a probe object; Can delete, the destination allowed the probe object to be deleted; Sealed, sealed 7 records at 1580 bytes to the canary cell; Signatures verify, the root and shard manifest signatures verified and the operational recipient wrap opened; RUNLOG fresh, the RUNLOG entry verified fresh and chained; Bytes intact, 7 records decrypted and matched the known corpus byte for byte; Restored, 7 records restored over the isolated canary cell; and Restore verified, every restored byte matched the known corpus exactly.

The known corpus is deterministic

The data a flight seals is the known corpus in corpus.ts. It is not random and it does not read a clock. It is generated in code from constant seeds, so the bytes are identical on every flight and on every deployment. That determinism is the whole point: the known answer is exactly known, so any deviation of even one bit is provable.

The corpus is exactly seven records, chosen to span the byte shapes that catch real corruption.

Record Shape What it catches
canary/ascii A plain ASCII sentence A basic round-trip and text mangling
canary/utf8 Multibyte UTF-8 with accented Latin, an emoji and a CJK ideograph An encoding-layer mangle of multibyte characters
canary/all-bytes Every byte value from 0x00 to 0xFF A byte that a transport or codec cannot carry cleanly
canary/zeros A run of all-zero bytes Padding or truncation that hides in zeros
canary/ones A run of all-0xFF bytes A stuck or flipped high bit across a block
canary/json A small structured JSON object A serialisation or framing error
canary/block-1k A kilobyte from a deterministic generator A silent transposition or truncation anywhere in a larger block

The kilobyte block is deterministic, not cryptographic

The kilobyte record is filled by a tiny deterministic generator seeded from a constant. It is explicitly not a cryptographic source of randomness. Its only job is to be non-trivial yet reproducible, so a quiet truncation, padding or transposition in the archive path shows up as a byte delta rather than hiding in a block of zeros.

The known bytes and the corpus total live only in the engine code. They are never shipped to the browser and never sent to the vendor. The console sees liveness and counts, not the corpus.

Integrity is a byte-exact known-answer compare

Aspect 6, decrypt-integrity, is where the canary earns its keep. After the read-back verifies the signatures and the freshness, the flight decrypts every record and compares it to the known corpus. There are two layers of checking here, and the second is the stronger one.

The first layer is the archive’s own integrity. The reader in reader.ts recomputes each record’s plaintext SHA-384 as it reassembles it, so a record that decrypts to anything other than what was sealed throws. That catches a flipped bit in a stored segment, a tampered manifest, or a corrupted RUNLOG, each of which maps to the right dead aspect.

The second layer is the known-answer compare. Even an archive that is internally self-consistent can still hold the wrong data. So the flight matches each decrypted record against the known corpus by exact bytes. The match is by value, not by name (record names are stored only as privacy-preserving hashes), so it is order-independent. A known record that never appears, or an unexpected record that appears, both count as a drift. The dead reason then reports precisely how many records diverged and how many bytes strayed.

Why the known-answer compare is the strongest check

A canary that trusted only the archive’s own hashes would call an internally-valid archive of the wrong data alive. The engine’s validator seals a corpus with one byte of one record changed and proves the byte-exact compare against the known data still calls it dead, where the internal hash alone would not. That is the difference between proving an archive is self-consistent and proving it holds exactly the data that was sent.

Isolation: a pure key-rewrite namespace

A flight must exercise the real seal, read and restore code, but it must never collide with a real archive. That is what PrefixedDestination in prefixed-dest.ts does. It wraps a real destination and prepends a fixed prefix, _CANARY/runs/<runId>/, to every put, get, conditional-put, delete and list. The seal pipeline and the reader address objects by their ordinary logical keys, and through this wrapper those keys land under the canary prefix instead.

The wrapper is a pure key rewrite. It adds no behaviour of its own, so the canary runs the identical code paths a real backup uses, only namespaced. That is what makes the proof meaningful: it is the real path, not a stand-in. Because every object a flight writes lives under its own run prefix, a whole flight’s residue is one prefix to delete, and it can never land in the same place as a customer archive (which lives under different prefixes). The restore aspects write back into that same isolated cell, never a real or production binding.

Deeper detail: object-lock status forwards unchanged, and the read-side store

Object-Lock configuration is a property of the whole bucket, not of a key prefix, so the wrapper forwards an object-lock-status probe to the inner destination unchanged. When the inner destination cannot run the probe, the wrapped destination honestly reports “unknown” rather than a fabricated verdict. The wrapper’s list also strips the namespace prefix back off the returned keys, so the caller sees logical keys, mirroring how the prune lists run trees. On the read side, the canary adapts the destination into the reader’s object store with the same prefix; a missing object throws, so an absent canary object fails the read-back loudly rather than verifying nothing.

Ailing is not death, and a delete refusal is a note

The aspect outcomes carry a deliberate distinction the rest of the system depends on. A death (fail) is detected drift on the known answer. Ailing is a fault that stopped the check completing, and it is reported as a status, not as a failed aspect.

A configured destination that cannot be read at all reports pending rather than crying wolf before setup. A write probe that fails halts the flight as ailing, because an unreachable path is not proven drifted. A delete that an immutable bucket refuses is recorded as a note on aspect 2, never a failure, because lacking the delete capability on an immutable destination is a posture observation, not a dead canary. A signature, freshness or integrity failure on the canary’s own freshly-written archive, by contrast, is a real drift and is reported dead.

A delete refusal is informational, never a failure

The delete-probe is a coarse observation. A refused delete only suggests immutability; it is noted, not failed. The authoritative immutability signal is the posture immutability check, fed by a real capability probe of the bucket’s Object-Lock configuration. The posture does not infer immutability from this delete behaviour. See immutability and attestation for that check.

Every posture runs all eight aspects

The read aspects need a key that opens the canary’s own cell. Where an operational key is present the flight opens through the recipient capsule, so the wrap decapsulation is exercised on every flight. Where the engine holds no in-account read-back key, the flight opens its own cell with the per-run master it just sealed under, so the full eight-aspect cycle runs and reports alive on a clean flight.

Neither path exercises the break-glass wrap. The engine has never held that private key and cannot, so no running engine proves the offline key opens what was sealed to it. That property is proven by attended verification, covered in break-glass and offline recovery.

How the engine proves each failure maps to the right aspect

The aspect-to-failure mapping is not asserted by hand; it is tested. The engine validator at test/validate-canary.ts seals the corpus through the real pipeline and then injects one fault at a time, proving the canary catches each and maps it to the right dead aspect: a flipped data bit goes to decrypt-integrity, a tampered manifest to read-signature, a corrupted RUNLOG to runlog-freshness, and an internally-valid archive of the wrong data to the known-answer compare. A clean flight matches every known record byte-for-byte and reads alive.

This is what lets the canary’s verdict be trusted: every detection mechanism genuinely fires on a real fault, so the bird cannot silently fake a pass and will go dead on a real issue. There is no console-side canary validator; the proof lives in the engine. The “Preview a death” button on the canary screen is a pure client-side simulation that never calls the engine, so do not read it as a real flight result.

Where this fits

For the canary concept, the five liveness states and how to read the screen, start with the canary. For configuring it, choosing its destinations, the cadence and the alert behaviour, read configure and operate the canary.

The integrity chain a flight reads back (the signatures and the RUNLOG) is described in verify at seal, and the underlying signing and encryption in cryptography. For the keyed drill that proves a real backup is recoverable, which the canary deliberately does not do because it carries synthetic data, read prove recoverability. For the no-custody trust model that the engine-side byte comparison rests on, read the no-custody trust model.

Last updated .