What downpipes alerts you about: events and severities

This is the reference for which events your downpipes engine can notify you about. It is authored from the engine’s event model and its actual emit sites, not from the console rule form, because the two do not match exactly: several events the engine emits are reachable only through a rule set to all events. This page is the honest catalogue for the self-hoster wiring up alerting.

The event model defines twenty-three events, and all twenty-three now fire from a real code path. Until recently restore-applied did not: it was selectable in the rule form, passed validation, and nothing emitted it, so a rule that named it matched nothing. That is fixed, and the table below is the full set.

The events the engine genuinely emits

These twenty-three events each have a real emit site in the engine. The severity column is the engine’s fixed mapping (severityOf, engine/src/notify-routing.ts); posture-regression and run-at-risk-eviction are the events whose severity can be raised to critical by the emitting path (posture-regression when the failing check is critical; run-at-risk-eviction when eviction would leave only one proven copy).

The emit sites are spread across the engine by concern. The backup-failure and backup-stale alerts fire from runAlertPass in engine/src/cron/alert-passes.ts, called each tick by the reconciliation cron in engine/src/cron/drive.ts; the same pass also builds the backup-success emission from the newest resolved run, via reconcileAlerts (assembled onto the scheduler DO from engine/src/sched/scheduler-do-sre-alerting.ts), so a manual run’s success is picked up by the same sweep as any scheduled run. The only backup-success site in engine/src/admin/router-ops.ts is the synthetic “Send Test” route, which is unrelated to real runs. The credential-expiry alert fires from runExpiryPass, also in engine/src/cron/alert-passes.ts. The backup-volume-regression alert fires from emitVolumeRegressionAlerts in engine/src/cron/retention-pass.ts, and the source-detached alert fires from the scheduled source-drift pass in engine/src/cron/alert-passes.ts; both are edge-triggered so a persistent condition pages once, not every tick. The restore-test events fire from engine/src/cron/restore-test-pass.ts, and a read-back verification failure at seal maps to restore-test-fail and is fired from engine/src/seal/verify-at-seal.ts. The canary and update events fire from engine/src/cron/notify-passes.ts. The posture-regression, recovery-code and dual-control-disabled alerts are routed by engine/src/admin/router-notify.ts, and the role-change and auth-credential-change alerts fire from engine/src/admin/router-rbac.ts and engine/src/admin/scim.ts (role-change on a role grant or change, a member removal, and a SCIM offboarding) and from engine/src/admin/router-account-session.ts and engine/src/admin/router-auth-flow.ts (auth-credential-change on a passkey-credential revoke and on a recovery-code regeneration). The replication-degraded and run-at-risk-eviction alerts fire from reconcileReplicationAlerts (assembled onto the scheduler DO from engine/src/sched/scheduler-do-sre-alerting.ts, the same mixin that builds the backup-success emission). The cron entry points re-export from engine/src/index.ts.

Four of these events also fire when the condition clears

The table above says when each event fires. For four of them the same event name fires a second time when the condition goes away, so a channel that opened an incident can close it rather than leaving it open until a human notices.

The four are backup-failure, backup-stale, replication-degraded and run-at-risk-eviction. Each recovery reuses the identical event name and downpipe as the alert that opened it and carries a recovered marker, and it reuses the severity the trigger fired at, so a recovery clears the same minimum-severity bar its trigger crossed and reaches the same rule (buildRecoveryEmission and the replication branches in engine/src/sched/scheduler-do-sre-alerting.ts). It is emitted on the falling edge, once, and a recovery whose delivery fails is retried within a bounded window so a transient problem cannot strand an incident open.

None of the other events on the table clears this way. Where a recovery matters for one of them the engine gives it a name of its own, which is what canary-recovered is: a distinct event, not a marked repeat of canary-dead. So a rule that selects events by name and wants the close as well as the open needs no second entry for the four above, and does need canary-recovered alongside canary-dead.

What a channel does with the marker depends on the channel, and incident channels sets out which kinds close an incident and which are fire-only by design.

Sign-in from a new location (opt-in)

One event is off unless you ask for it. The sign-in-new-context alert is a “did you just sign in from a new place?” signal that helps a human spot a takeover, and it is off by default because a new location is not necessarily an attack, so a noisy default would train people to ignore it.

An Owner turns it on with the “Notify on a sign-in from a new location” control in the Security centre. Once on, a successful sign-in fires the event when it comes from a coarse network context, an IPv4 /24 or IPv6 /48 prefix, that is not in the account’s recent per-operator baseline. It compares against a bounded, coarse seen-set held in your own account, so it stores no raw IP address and keeps no location history. The alert never blocks or delays the sign-in, it is a fire-and-forget notification sent after the sign-in has already succeeded, and its detail is a generic one-liner that carries no IP, prefix, email, subject or place, so it is safe to route to a shared channel. Because it rides the normal event model, it is delivered by any rule whose events are set to all, at warning severity or lower.

Picking events by name, and when to use an all-events rule

The console’s rule form and the engine’s rule validator now cover the same closed set, so every event on the table above can be selected by name. The engine’s validation set is built directly from NOTIFY_EVENT_NAMES (NOTIFY_EVENTS, engine/src/notify-routing.ts) so the two cannot drift apart, and the console’s checklist carries all twenty-three of the same names with the severity each one fires at (NOTIFY_EVENTS, console/src/screens/notifications/shared.ts).

That was not always so. The checklist held twelve entries until 4 August 2026, which left eleven events, among them backup-volume-regression, source-detached, canary-dead and update-rollback-needed, deliverable only by a rule whose events field was set to all. If you built a rule during that period from the checklist, it carries the events that were selectable then and nothing else, so re-open it and tick the ones you want rather than assuming the new entries were added to it.

An all-events rule is still the right choice when you want everything the engine can emit, present and future: it needs no revisiting when a new event name is added, whereas a hand-picked list does. A named checklist is the right choice when you want a channel to stay narrow. The minimum-severity filter applies either way, so an all-events rule at critical carries backup-failure, restore-test-fail, posture-regression, recovery-code-abuse, canary-dead and update-rollback-needed, and warning picks up the rest.

The restore-test-pass subtlety

A restore-test-pass alert does not always mean a drill verified your data. The scheduled restore test emits this info-class event in two degenerate cases where no real verification happened, and it distinguishes them in the detail string rather than failing or staying silent (runScheduledRestoreTest, engine/src/index.ts).

The reason the two degenerate cases emit info rather than a critical fail is that neither is a failure of the downpipe: a break-glass-only posture is a deliberate choice, and a downpipe that has not run yet is simply new. But neither proves recoverability, so read the detail string before treating a restore-test-pass as a verified drill. A genuine failure of the in-account test, by contrast, emits restore-test-fail at critical with the coarse reason.

Defaults, in one place

Failure and stale alerts are on by default once you add your first channel: the engine auto-creates a default global rule selecting backup-failure and backup-stale, but only when it is the first channel and no rule exists yet. Backup-success is off by default in the sense that this auto-created rule never selects it, so a run’s success reaches no channel until you write a rule that includes it. Once such a rule exists, its digest setting decides how backup-success is delivered, not whether it is delivered: a digest of “off” (the rule form’s default) sends it immediately, and only setting the digest to daily or weekly batches it. Everything else reaches a channel only through a rule you write, either by name or with an all-events rule.

Where this fits

To add channels and write the rules that route these events, read the notifications guide. To see the run outcomes behind backup-failure, backup-stale, and the restore-test events, read the Runs activity view. The canary that drives canary-dead and canary-recovered is described in the canary reference, and the posture checks behind posture-regression in the posture score. A role change reaches any rule that selects role-change or all events, and it is recorded in the audit log as well. An applied restore now does both: it reaches any rule that selects restore-applied or all events, and the same apply is anchored in the audit log by its signed receipt.

Last updated 4 August 2026.