Configuring, operating and watching the canary
You can disable the canary, repoint it to different destinations, or change its cadence, and an owner can do all three from the console with no command line. This page also covers flying it on demand and reading its alerts. What the canary is and what one flight proves are covered on the canary and the eight aspects; this page is about running it day to day.
It is written for a self-hoster who operates the account and for on-call, who need to know what a canary page means and trust that a destination that stays dead pages once, not every hour. Everything here is taken from the canary routes in the engine, the scheduler durable object, the notification module and the canary screen in the console.
Who can do what
The canary’s controls are split across two permission levels, and the engine is the enforcement point. Reading the canary view is available to any authenticated role, because it is liveness and counts, not a write and not a secret. Flying the canary now is gated on run.trigger, the same capability that triggers a backup, so operators and above can do it. Changing the configuration is owner-exclusive.
| Action | Who can do it | How it is gated |
|---|---|---|
| Read the canary view and history | Any authenticated role | No per-route capability check; any authenticated caller |
| Fly the canary now | Operator and above | run.trigger, the backup-trigger capability |
| Enable or disable it | Owner only | An inline owner check in the router, re-checked in the durable object |
| Repoint which destinations it flies to | Owner only | The same owner check and re-check |
| Change the cadence | Owner only | The same owner check and re-check |
There is no separate canary-configuration capability. The router gates the configuration route on the caller being the owner, and then the durable object re-resolves the caller’s role from its own tables and refuses a non-owner, so a router bug cannot let a non-owner change the configuration. In the console, the owner controls render disabled with a reason for everyone else rather than being hidden, so a non-owner can see that the controls exist and why they cannot use them.
Fly-now is not owner-only
A common mistake is to assume only the owner can fly the canary. Flying it now is an operator-and-up action gated on run.trigger. Only enabling, disabling, repointing and changing the cadence are reserved to the owner.
Configuration choices
The canary is on by default. An account that never touches its settings runs the on-by-default bird against its destinations, up to the twelve-destination cap described below. There are three things an owner can change, all on the canary screen under Govern.
Enable or disable. Turning the canary off stops the proof that each destination can be written, sealed, read, restored and verified byte-for-byte between real runs. Your scheduled backups keep running unchanged, but you lose the early warning, so the console asks you to confirm a disable and explains exactly what you give up.
Which destinations it flies to. By default the canary flies to your configured destinations and auto-includes any you add later. An owner can instead pin an explicit subset. Pinning to a subset means a destination you add afterwards is not flown until you add it to the set. An empty selection is treated as the all-destinations default rather than flying to nothing, and a pinned destination id that does not name a real destination is refused.
A flight covers at most twelve destinations, and the rest are silent
One flight is capped at twelve destinations (CANARY_MAX_DESTS, engine/src/canary/types.ts), so a worker invocation cannot be blown by a large collection. The list is ordered the same way every time, so destinations past the twelfth are never flown, not on that flight and not on a later one. They also do not appear on the canary screen: the engine records the exclusion with their labels into the fault ledger, and that record rides in a support bundle rather than in the console view. If you hold more than twelve destinations, read the canary’s aggregate as covering twelve of them, and pin the canary to the ones you most need proven rather than leaving it on the default.
The cadence. The default interval is 3600 seconds, every sixty minutes. When you set a custom interval the engine clamps it to a floor of 300 seconds and a ceiling of one day, so a misconfiguration cannot ask for a flight every second or once a decade.
The cadence floor is 300 seconds, not sixty minutes
The smallest interval the engine will accept is 300 seconds, which is five minutes. The default of 3600 seconds is the recommended cadence, not the floor. Whatever interval you set, a flight rides the cron rather than firing on the exact second (see below), so the effective cadence is the interval rounded up to the next cron tick.
How the cadence actually lands
The canary does not fire on its own timer. It rides the engine’s existing fifteen-minute cron. On each cron tick the engine asks the durable object whether the canary is due, and if it is, the object allocates the flight and the engine flies it. So a flight on the default interval lands within fifteen minutes of its mark, which is why the honest description of the default cadence is on by default and roughly hourly within a fifteen-minute window. A flight already in progress is not re-flown while it holds its in-flight lease, and a lease left stale by an evicted worker is reclaimed on the next due check, so an interrupted flight never wedges the bird.
Flying the canary now
Flying it now is for when you have just connected a destination, rotated a credential, or want to watch the bird before relying on a destination. The “Fly the canary now” button on the canary screen calls the engine, which arms the bird as due in the durable object and then drives the flight in the background of that same request, so a fresh result lands within moments rather than waiting for the next cron tick.
Open the canary screen
Go to the Canary screen under Govern. If you are an operator or above, the “Fly the canary now” button is enabled; otherwise it is shown disabled with the reason.
Fly it
Press the button. The bird takes off immediately in the illustration, and the console polls the engine while the flight runs so you watch it land alive or die. The flight runs the whole eight-aspect cycle against each destination the canary flies to, in turn.
Read the result
When the flight settles, the screen refreshes with the per-destination result and the eight-aspect proof. A death names the failed aspect and the byte count. Outside a manual flight, the canary flies roughly hourly on its own because it rides the cron.
How the alerts behave
The canary uses two notification events, and they are designed so a real fault is loud once and a persistent fault does not become hourly noise.
| Event | Severity | When it fires |
|---|---|---|
canary-dead |
Critical | A destination’s canary transitions into dead: a byte strayed from the known data. Page now. |
canary-recovered |
Informational | A previously-dead destination’s canary flies clean again. |
The load-bearing property is that both fire only on a per-destination transition. A destination that stays dead does not page you every flight; it pages once, on the transition into dead, and again only if it recovers and later dies again. A recovery sends a single informational notice. Because each destination has its own liveness, a death on one destination pages for that destination while the others stay quiet. The alerts route through your existing notification channels and rules, the same path every other engine notification uses, so where a canary page lands is whatever you configured under notifications.
What a canary alert can and cannot say
A canary alert and the console detail name the failed aspect and the byte count and nothing more. The detail is redaction-safe by construction: it never carries a key, a value, an object key, an endpoint, a bucket or a region. A typical dead detail reads as the named destination, the failed aspect, and how many bytes strayed from the known data. For the full alert and channel model, see notifications and channels.
What is and is not in the audit log
The canary writes to two different places, and the distinction matters for an auditor reading the chain.
A configuration change that is applied appends a canary-config entry to the tamper-evident audit chain: who made the change and when. The entry records that the configuration changed, never the new value, so the chain shows that an owner repointed or re-cadenced the canary without leaking the setting itself.
The hourly flight results are not written to the audit chain. They live in a bounded in-durable-object history ring that holds the last forty-eight flights, which is two days of hourly flights. The ring is what the canary screen’s recent-flights list reads, and it is capped so the object’s storage stays bounded, mirroring the audit and notification history rings. So if you are looking for evidence of a specific past flight beyond the last forty-eight, the audit chain will not have it; the chain carries the configuration changes, not the flight-by-flight outcomes.
Hourly results are not audit-chain evidence
Treat the flight-history ring as recent operational state, not as a long-term evidence record. For a dated, signed evidence artefact of the system’s behaviour, generate a signed report; for the immutable record of who changed what, read the audit log.
Previewing a death without causing one
The canary screen offers a “Preview a death” button. It is a pure client-side simulation: it lets an operator watch the bird take off, fail and die on fabricated data, so they know what a real death looks like on the screen, without waiting for or causing a real one. It never calls the engine and never touches the live canary or its history. The simulated detail is shaped to look exactly like a real death (the path works up to the integrity check, then a few records diverge), but it is an illustration, behind an unmistakable “simulated preview” banner with a way out.
Where this fits
For what one flight actually checks, aspect by aspect, read the eight aspects. For the canary concept, the liveness states and the multi-destination model, read the canary.
For the notification channels and rules a canary alert routes through, read notifications and channels and alert events. For how the destinations the canary flies to are configured in the first place, read multiple destinations. For the roles and capabilities behind who can fly it and who can configure it, read roles and capabilities.
Last updated .