Skip to content
downpipes docs

Account compromise: the eviction and recovery runbook

This is the runbook for the worst case: your own Cloudflare account is presumed to be in hostile hands. It is for the responder who has decided this is no longer an ordinary failed backup but a compromise, and who needs to capture clean evidence and recover in the right order without believing in a containment the product does not provide.

The honest framing comes first, because it changes how you work every step below. There is no one-button account freeze in downpipes; it is not built. An attacker who controls the account can undo anything you set there. So the early steps are about keeping your evidence clean and your archives provably untouched, not about locking the attacker out. The property the whole runbook leans on is simpler and stronger: the archives are ciphertext in your own bucket, the break-glass private key is offline and was never in the account, and the offline reader restores with neither the engine nor the vendor in the loop. An incident can cost you assurance. It should never cost you recovery.

If you are not yet sure this is a compromise rather than a misconfiguration, start at incident response and its first-five-minutes decision tree, then come back here once you have a reason to believe the account itself is hostile.

What an account attacker can and cannot do

Before you act, know the stakes precisely, because they decide which keys you rotate and what you can still trust. The threat model is explicit about an adversary who holds the Cloudflare account.

The attacker may hold What that lets them do What it does not let them do
The signer private (it lived in the account) Forge runs that verify under your pinned signer, by rewriting a run’s bytes, recomputing its hashes, rebuilding the manifest, and re-signing the root, so verify alone is no longer a trustworthy gate for runs they re-sign Roll the run log back past your recorded --min-runlog-index pin without that rollback showing, or reach the offline break-glass key
The operational private, in the two-recipient posture Read your archives through the operational decryption path Touch the offline recovery path, which never used that key
Write access to the destination bucket Delete objects, which surfaces loudly Forge a run the verify accepts; deletion is the only move

They have never held the break-glass private key, because it was generated in the browser, downloaded to your machine, and never transmitted to the engine or the vendor. That is the design, and it is why recovery stays yours through a full account compromise. Where an operational key has been added, the engine itself holds a decryption-capable key, so an account attacker can read your data through that path. Switching to the strict break-glass-only posture removes that operational private from the engine, so there is no decryption-capable key left in the account for an attacker to take. See recovery postures for the trade-off and how to switch.

Do not rotate keys before you have captured evidence and verified the archives

Archives sealed before a rotation open only with the key that sealed them. Rotating the break-glass recipient while you are still unsure about custody converts an incident into permanent data loss. Rotation has a place in this runbook, but it comes after the evidence is captured and the archives are verified, and it is always done knowing you keep the old key for the old archives.

The eviction flow in order

Work these in order. The order is what makes the evidence defensible and keeps a confused rotation from destroying the very thing you would recover from. Do not “clean up” first.

  1. Freeze writes (for evidence, not containment)

    Pause every downpipe (the Downpipes screen, enabled off, or POST /admin/downpipes per pipe). If you control deploys, remove the */15 entry from [triggers] in wrangler.toml and redeploy so the reconciliation tick stops driving new dispatch.

    Be honest about the mechanics. Pausing stops new dispatch only. A sliced run already in flight continues on its own seal alarms until it finalises; it does not stop the moment you pause. And an attacker with account control can undo anything you set here. This freeze reduces noise in the evidence you are about to capture. It is not containment. Containment, to the extent it exists at all, is the credential rotation later in this flow.

  2. Capture evidence off the account, before anything else changes

    Export and store these somewhere the account cannot reach. Capture them now, before any rotation, because rotation changes what the account looks like.

    Export Endpoint Why it matters
    The audit log GET /admin/audit/export (JSON) Record its headSeq and headHash off-account. The chain is tamper-evident, and the head hash you hold is what makes a later rewrite provable.
    A support bundle GET /admin/support/bundle Versions, provenance, the preflight report, and coarse run rows, in one redaction-safe document.
    The config history GET /admin/config/history The signed config version history with the chain head. Pull GET /admin/config/version?id=N for any version you may need to argue from.

    The single most important thing here is the head sequence and head hash. The audit chain is hash-linked, so each entry commits to the one before it. If you record the current headSeq and headHash somewhere the account cannot touch, then any later attempt to rewind or rewrite the chain becomes provable: a re-run of GET /admin/audit/verify against a tampered chain will not reconcile to the head you hold. Without that recorded head, a rewind is far harder to argue. The audit export and the support bundle carry operator identity, including emails and source addresses, so treat them as sensitive operational records and store them accordingly; they are tamper-evident, not anonymous.

  3. Verify the archives from a clean machine

    The destination bytes plus your offline keys are the trust anchor now, not the account. On hardware the account never touched, with the open-source downpipe CLI and your pinned signer.pub, verify a run:

    downpipe verify --run <runId> \
  --archive <dir> \
  --identity identity.key \
  --signer signer.pub \
  --check-bundle \
  --receipt receipt.json \
  --min-runlog-index <n>

    Swap --archive <dir> for --s3-endpoint <url> --s3-bucket <name> if you are verifying straight from the destination bucket rather than a local copy.

    Pass --min-runlog-index <n> from the pin recorded on your recovery sheet. What this proves, all at once: the signed root verifies under your pinned signer, every hash recomputes, completeness holds across the manifest, and the run log has not been rolled back past your pin. A bucket-write adversary cannot forge a run this check accepts. They can only delete objects, and a deletion surfaces loudly rather than passing as a valid run. The console Keys screen prints the exact invocation for your deployment.

    There is also a keyless attestation, POST /admin/restore/attest, that checks the signature, completeness and anti-rollback without a key and without touching data. It is a useful in-account cross-check, but treat it as an engine-side assertion. It is not a Cloudflare-side verdict, and during a presumed compromise the engine and the account are exactly what you have stopped trusting, so the clean-machine downpipe verify against your pinned signer is the verification that counts.

  4. Rotate every Cloudflare credential and the destination keys

    This is the closest thing to containment, and it is Owner-gated. In the Cloudflare dashboard, work through the account itself.

    1. Remove or re-verify every account member, and force re-authentication across the account.
    2. Rotate every API token. A standing deploy token should not exist at all; if one does, that is its own finding to fix. See Cloudflare token scopes for the minimal, short-lived posture.
    3. Revoke Cloudflare Access sessions, then re-check the Access policies and the identity provider behind them.
    4. Rotate the destination credentials at the destination provider. For an S3-compatible destination, rotate DEST_ACCESS_KEY_ID and DEST_SECRET_ACCESS_KEY there, because the attacker read them from the account.

    In the engine, terminate every native session, then kill the bootstrap bearer if it is still live. Revoking Cloudflare Access in step 3 does not touch the engine’s own sessions: the engine accepts a verified Cloudflare Access JWT as its highest-precedence credential but falls through to its own native session cookie (and, unless disabled, the ADMIN_TOKEN bearer), so cutting the Access JWT leaves a passkey, recovery-code, OIDC, or SAML cookie session live. Use the Owner-only Security Centre “terminate all sessions”, which deletes the engine’s session signing key so every outstanding native session fails its next check and an attacker riding one is evicted. Recovery codes are not affected. They used to be, because the codes were verified under the same key the action deletes, but the recovery hash key is now a separate record that is materialised before the session key is deleted, so a printed recovery sheet keeps working across a sign-out-everyone. Do not regenerate the codes as part of this step: regenerating replaces the working set, and the sheet an Owner is holding is exactly what you need if the attacker also took a passkey. Regenerate only when you have reason to believe the codes themselves were captured, and only once you can print and distribute the new sheet. For a single named principal rather than a global sign-out, use the per-user terminate-user control instead. Both of those controls, and retiring the bearer below, are step-up gated: each asks for a fresh passkey assertion from the person running the eviction. Have your own authenticator to hand before you start this step, because it is the one step in this runbook where a missing authenticator stops the eviction rather than delaying a convenience. A Cloudflare Access session and a bare break-glass token session are exempt, so if you have already cut Access in step 3 you are on a cookie session and will be prompted. Then handle the bearer: if you trust the engine enough to drive it, retire the token in-app (Security Centre, Retire break-glass token); otherwise delete the secret with your rotated operator credentials (npx wrangler secret delete ADMIN_TOKEN). A live shared bearer is one human holding two identities, which quietly defeats dual control, so it has no place in a hardened account.

  5. Re-stand-up clean, or recover in place

    Now choose a path. Both keep the old archives readable with the old break-glass key; say so plainly in your incident record either way.

    The cleaner path is to re-stand-up in a fresh account. Deploy the engine and console from a release whose artefact digest you verified against the published checksum. Run a new key ceremony (a new break-glass pair, a new signer, fresh recovery codes, a new printed recovery sheet). Configure a destination you control, bind the sources as they exist in the new account, and resume schedules. Keep the old bucket (or copy its bytes out) and keep the old identity.key and old signer.pub for as long as those archives matter, because old archives open only with the old break-glass key and verify only under the old pinned signer.

    The other path is to recover in place after the rotation above. Redeploy the engine from a verified release into the rotated account, re-run the ceremony treating every in-account secret as burnt, re-apply secrets, and resume schedules. Then run a drill and a scheduled restore test before you trust the pipeline again.

The signer-presumed-lost branch

The signer private lived in the account, so a full compromise means you should presume it lost. Choose honestly between two branches, and write down which you chose and why.

The defensible default is signer presumed lost. Rotate SIGNER_PRIVATE through the ceremony so future archives sign under a new key. Old archives still verify, because the reader verifies each run against the signer you pin for it: the operator pins both, the new key for new runs and the old signer.pub only for genuinely-old runs captured before the compromise. That last qualifier matters, because a stolen signer can re-sign a tampered run under the old key and it will still pass verify against the old pin, so the old pin is a trust anchor only for runs you held before the account fell. Re-print the recovery sheet with the new fingerprints, retain the old sheet with the old pins, keep the --min-runlog-index pin current, and lean on the out-of-band audit head hash you recorded at the start to bound which runs are genuinely yours. A stolen signer can forge plausible-looking runs that verify until your rotation lands, so treating the signer as burnt and rotating it promptly, backed by the run-log index pin, is what bounds the window.

Keeping the signer is only defensible if the evidence shows the secret was never readable, which is rare in a full compromise. If you do keep it, record why in the incident notes rather than leaving it implicit.

Below threshold, in-account restore still works

If a break-glass custodian is unavailable and you are below your Shamir threshold for the offline key, you are not stranded while the account and the operational key still live. In-account restore continues to work in that window. The break-glass key is the path that survives losing the account; it is not the only path while the account is healthy. The engine and the vendor cannot reconstruct the break-glass key for you, by design.

Who to page and what they need

A compromise needs the right people awake with the right credentials. Plan this before the incident, not during it.

Who When they are needed What they need
The Owner Every authority action here is Owner-gated: roles, retiring the bearer, minting or revoking support credentials, the config approval policy, and the dashboard rotation step Their passkey or a recovery code, plus Cloudflare dashboard access
An Approver, or a second Owner Any restore apply is dual control: maker is not checker, single use, with a 24-hour expiry Their own sign-in; a recovery code gets a sole survivor signed in but never lets one person approve their own apply
The break-glass custodians Offline recovery, or the clean-machine verify when run with the identity key The identity.key, or enough Shamir shares to meet the threshold, and the printed recovery sheet

Fix the topology before an incident, not during one. The Security Centre warns you ahead of time: a two-owners finding when you have a single Owner, a dispose-bootstrap-token finding while a live bearer remains, and a recovery-codes-low finding when your remaining codes run low. They are the same lesson said three ways, because dual control quietly fails when one person can present two identities.

Communication discipline

The audit chain is the timeline of record. Every privileged action and every denied attempt during the incident lands in it, hash-chained, including the eviction steps themselves: role changes, the bearer retire, credential grants and revokes, restore requests and approvals. Run the incident through the product wherever the product can see it, and narrate the rest, such as dashboard actions and provider tickets, in your own notes with timestamps.

At the end, verify and seal the record. Run GET /admin/audit/verify, then export the chain again with GET /admin/audit/export and record its head hash out of band. Attach that closing export beside the evidence you captured at the start, so the post-incident review argues from the tamper-evident record rather than from memory.

Why deletion is loud but forgery is not possible

A bucket-write adversary, the threat model’s destination attacker, can only delete or corrupt objects on the destination. They cannot construct a run that downpipe verify accepts, because the run carries a signed root over a manifest of content hashes, and the verify recomputes every hash and checks the signature against the signer you pin. A tampered object fails its hash; a missing object fails completeness. Both surface as a loud verification failure rather than a quietly accepted bad run.

That asymmetry is why the early discipline is so strict about leaving the bucket alone. The bucket is both your evidence and your recovery source. The one thing an attacker can do to it, deletion, is also the one thing your own careless “tidying” would do, and the verification path cannot tell a hostile deletion from an accidental one. Capture, verify, then decide deliberately what to do with the bucket.

Where this fits

For what to expect from a key rotation outside an incident, whether you are rotating the break-glass key, adding an operational key, or re-keying completely, see rotating your keys. For the calmer first-response decision tree, and how to tell a configuration fault from a genuine compromise before you start here, see incident response. For the posture choice that decides whether the engine holds a decryption-capable operational key, and how the break-glass-only posture removes it, see recovery postures. For the key material you rotate and the ceremony that mints fresh keys and a new recovery sheet, see the key ceremony and recovery kit. For the offline verify and restore that keep recovery vendor-independent, see break-glass offline recovery. For the minimal, short-lived token posture that means there is no standing credential for an attacker to find, see Cloudflare token scopes.

Last updated .