Incident response and the first-five-minutes decision tree

This is the runbook for the moment something is wrong with a live downpipes deployment and you need to act without making it worse. It is written decision-tree first, because the most valuable thing at 3am is knowing which of a few coarse classes you are in and what to read next, not a wall of detail. The detail follows once you have placed the incident.

Hold the standing facts that make every branch survivable. The archives are ciphertext in your own bucket. The break-glass private key is offline and was never in the account. The offline reader restores with neither the engine nor the vendor in the loop. An incident can cost you assurance, and it should never cost you recovery, so the discipline below is about preserving evidence and recovery, not about heroics.

The coarse error classes

When the Runs screen shows failed rows, read the coarse error class on the failure and place it. There are three for a failed backup, and which one you are in tells you the first thing to read.

A CONFIG failure usually means a secret or a binding went away. The classic cause is a deploy that did not carry a console-attached source: wrangler.toml lists no sources, so a bare deploy drops them, which is exactly what deploy safety and bindings exists to prevent. The fix is to restore the missing secret or re-attach the source, and redeploy with npm run deploy if wrangler.toml changed.

An UPSTREAM failure is about reaching the destination bucket or reading the named source. The preflight destination item does a live read-only existence check on the RUNLOG key, so it tells you whether the destination a run would actually use is reachable and authorised. The fix is on the destination side (credentials or policy) or the source’s own state, not in the engine.

An ENTITLEMENT failure is a platform-level problem: a deploy the account refused, a cron that has stopped firing, or Durable Object errors. The account-side downpipe preflight and the in-engine preflight items for the Workers plan, the cron tick and the Durable Objects are where you confirm it.

The first five minutes

Read, in this order, before you change anything. Each step narrows the picture, and reading them in order is what keeps you from acting on a symptom before you have seen the cause.

1. GET /admin/status

   Read ready, the configuration-presence booleans, and the engineVersion. A boolean that was true yesterday and is false today is the incident. Status never returns a secret value, a fingerprint, a destination detail or a downpipe name, so it is safe to read and to share.

2. GET /admin/preflight

   Read which item moved off verified and what its evidence line says; the console Readiness card shows the same facts. Each item is probed live where the platform allows, so the evidence is an observed fact rather than an assumption, and the remediation names the Cloudflare product or onboarding step that covers it.

3. The Runs history

   Read GET /admin/history to see when the last good run finished, which downpipes are failing, and the coarse error class on each. This is what confirms the class you placed from the failure row and shows the blast radius.

4. The audit tail

   Read GET /admin/audit, newest first, for what changed, who did it, and by which method. The engine-observed events such as engine-version-change and engine-secret-present catch changes that no human admitted to, so the audit tail is where an unexplained configuration change shows up.

5. wrangler tail, only if you need live detail

   Run npx wrangler tail downpipe-engine if the four reads above have not localised it. The log lines carry coarse classes and cause digests, never values, so they tell you the shape of a live error without exposing data.

The diagnostic endpoints that exist

These are the read surfaces you will reach for, all admin-authenticated and all redaction-safe. Knowing the full set means you are never guessing whether a given fact is observable.

The audit trail and the support bundle carry operator identity, addresses and the coarse facts of an incident. That is deliberate: the trail is the timeline of record, and it is tamper-evident and hash-chained so a later rewrite is provable against a head hash you hold. Treat those exports as carrying personal and operational detail, store them somewhere the account cannot touch, and use the head hash as the anchor.

The disciplines that keep it from getting worse

Two reflexes do more harm than good in the first minutes of an incident, and the discipline is the opposite of the reflex.

Do not rotate keys mid-incident unless the old key is retained. Archives sealed before a rotation open only with the key that sealed them, so rotating the break-glass recipient while you are still unsure about custody converts an incident into data loss. Rotation has its place, but it comes later, in the eviction runbook, after evidence is captured and the archives are verified, and it is done knowing you keep the old key for the old archives.

Do not write to the destination bucket, and do not tidy its objects, while you diagnose. The bucket is both the evidence and the recovery source. The verification path recomputes everything from the bytes, so a deletion or a rewrite is the adversary’s move, not yours, and it destroys the very thing you would recover from. Leave the bucket alone until you have decided, deliberately, what to do with it.

Two more holds apply while you still need them. Do not delete the Workers yet; that is the offboarding flow, not the incident flow. And do not disable Cloudflare Access while you still need attributable sign-ins, because the attributable trail is part of what you are trying to preserve.

Telling a licence or update problem from a data problem

This is worth its own line because it is the fastest way to discard a whole category of false alarm. Neither the licence nor the update channel gates a backup or a recovery. So if backups are running and a restore proves out, a licence in a lapsed or community state and an update channel that is unconfigured are not your incident, however loudly they show in the console. Conversely, a genuine data-path failure will show as a failed run with one of the coarse classes above, or as a presence boolean that went false, not as a licence or update banner. Confirm the data path with GET /admin/status and the Runs history, and let the assurance surfaces be a separate, lower-priority question.

When to escalate to the eviction runbook

If the account itself is presumed compromised, stop working the symptom and switch to the eviction runbook. The signal for this is not a failed backup; it is a reason to believe the Cloudflare account is in hostile hands, for example unexplained member changes, sign-ins you cannot account for, or audit events with no human behind them.

Do not “clean up” first. The eviction runbook is worked in order: freeze writes to reduce noise in the evidence, capture the audit export and a support bundle off the account with the head hash recorded, verify the archives from a clean machine the account never touched, rotate every Cloudflare credential and the destination credentials, then either re-stand-up in a clean account or recover in place after rotation. The threat model is explicit about the stakes: an attacker with account control may hold the signer private and, in the two-recipient posture, the operational private, so they can read archives through the operational path and forge signatures on new runs. They have never held the break-glass private key, so they cannot take recovery away from you, which is the property the whole runbook leans on. The eviction runbook itself lives in the engine’s incident documentation.

Where this fits

This page is the canonical home for the first-response decision tree and the diagnostic read order. For the binding hazard behind most CONFIG failures and how to recover a dropped source, see deploy safety and bindings. For why the licence and the update channel never gate the data path, and the supervised-first stance on live apply, see upgrades and rollback. For the identity model behind the Owner-gated actions and the audit trail, see identity and access. For the property that makes every branch survivable, see the no-custody trust model.

Last updated 19 June 2026.