Account compromise: the eviction and recovery runbook

This is the runbook for the worst case: your own Cloudflare account is presumed to be in hostile hands. It is for the responder who has decided this is no longer an ordinary failed backup but a compromise. The responder needs to capture clean evidence and recover in the right order without believing in a containment the product does not provide.

There is no one-button account freeze in downpipes, and that changes how you work every step below. An attacker who controls the account can undo anything you set there. So the early steps are about keeping your evidence clean and your archives provably untouched, not about locking the attacker out.

The property the whole runbook leans on is simpler and stronger: the archives are ciphertext in your own bucket, the break-glass private key is offline and was never in the account, and the offline reader restores with neither the engine nor the vendor in the loop. An incident can cost you assurance. It should never cost you recovery.

If you are not yet sure this is a compromise rather than a misconfiguration, start at incident response’s first-five-minutes decision tree, then return once you have reason to believe the account itself is hostile.

What an account attacker can and cannot do

Before you act, know the stakes, because they decide which keys you rotate and what you can still trust. The threat model is explicit about an adversary who holds the Cloudflare account.

• The attacker may hold: the signer private (it lived in the account). Lets them: forge runs that verify under your pinned signer, by rewriting a run’s bytes, recomputing its hashes, rebuilding the manifest, and re-signing the root, so verify alone is no longer a trustworthy gate for runs they re-sign. Does not let them: roll the run log back past your recorded --min-runlog-index pin without that rollback showing, or reach the offline break-glass key.
• The attacker may hold: the operational private, in the two-recipient posture. Lets them: read your archives through the operational decryption path. Does not let them: touch the offline recovery path, which never used that key.
• The attacker may hold: the configuration keys (CONFIG_RECIPIENT_PRIVATE and CONFIG_WRAP_KEY). Lets them: open the sealed configuration export, and decrypt the credentials that the console stores. Does not let them: open an archive.
• The attacker may hold: write access to the destination bucket. Lets them: delete objects, which surfaces loudly. Does not let them: forge a run the verify accepts; deletion is the only move.

They have never held the break-glass private key, because it was generated in the browser, downloaded to your machine, and never transmitted to the engine or the vendor. That is the design, and it is why recovery stays yours through a full account compromise. Where an operational key has been added, the engine itself holds a decryption-capable key, so an account attacker can read your data through that path. Switching to the strict break-glass-only posture removes that operational private from the engine, so no key that decrypts an archive is left in the account for an attacker to take. See recovery postures for the trade-off and how to switch.

The eviction flow in order

Work these in order. The order is what makes the evidence defensible and keeps a confused rotation from destroying the very thing you would recover from. Do not “clean up” first.

1. Freeze writes (for evidence, not containment)

   Pause every downpipe (the Downpipes screen, enabled off, or POST /admin/downpipes per pipe). If you control deploys, remove the */15 entry from [triggers] in wrangler.toml and redeploy so the reconciliation tick stops driving new dispatch.

   Pausing stops new dispatch, and it stops the retention prune for the downpipes you paused, so the runs you are about to treat as evidence are not aged out from under you while you work. A sliced run already in flight continues on its own seal alarms until it finalises; it does not stop the moment you pause. And an attacker with account control can undo anything you set here, including the pause.

   This freeze reduces noise in the evidence you are about to capture. It is not containment. Containment, to the extent it exists at all, is the credential rotation later in this flow.

2. Capture evidence off the account, before anything else changes

   Export and store these somewhere the account cannot reach. Capture them now, before any rotation, because rotation changes what the account looks like.

   Export	Endpoint	Why it matters
   The audit log	GET /admin/audit/export (JSON)	Record its headSeq and headHash off-account. The chain is tamper-evident, and the head hash you hold is what makes a later rewrite provable.
   A support bundle	GET /admin/support/bundle	Versions, provenance, the preflight report, and coarse run rows, in one redaction-safe document. When the engine has a vendor support key (VENDOR_SUPPORT_PUBLIC), the download is sealed to the vendor and you cannot read it.
   The config history	GET /admin/config/history	The config version history with the chain head. Each entry is HMAC-chained under a key that stays inside the engine, not signed by the run signer. Pull GET /admin/config/version?id=N for any version you may need to argue from.

   The single most important thing here is the head sequence and head hash. The audit chain is hash-linked, so each entry commits to the one before it. If you record the current headSeq and headHash somewhere the account cannot touch, then any later attempt to rewind or rewrite the chain becomes provable: a re-run of GET /admin/audit/verify against a tampered chain will not reconcile to the head you hold. Without that recorded head, a rewind is far harder to argue. The audit export carries operator identity, including emails and source addresses, so treat it as a sensitive operational record and store it accordingly; it is tamper-evident, not anonymous. The support bundle does not copy operator emails, subjects or source addresses.

3. Verify the archives from a clean machine

   The destination bytes plus your offline keys are the trust anchor now, not the account. On hardware the account never touched, with the open-source downpipe CLI and your pinned signer.pub, verify a run:

   downpipe verify --run <runId> \
     --archive <dir> \
     --identity identity.key \
     --signer signer.pub \
     --check-bundle \
     --deep \
     --receipt receipt.json \
     --min-runlog-index <n>

   Swap --archive <dir> for --s3-endpoint <url> --s3-bucket <name> if you are verifying straight from the destination bucket rather than a local copy.

   Pass --min-runlog-index <n> from the pin recorded on your recovery sheet.

   Plain downpipe verify reads the signed root, the shard manifests and the run log only. It proves that the signed root verifies under your pinned signer. It proves that completeness holds across the manifests. The run log check proves that the run log has not been rolled back past your pin.

   It never reads the segment objects, so a changed or missing object passes it. With --deep, verify also decrypts every segment and recomputes every record hash, and it writes nothing. That is the object-level check, and it reads the whole archive.

   A bucket-write adversary cannot forge a run this check accepts. They can only delete objects, and a deletion surfaces loudly rather than passing as a valid run.

   The console Keys screen prints this command for a local or an S3/R2 archive, but without --min-runlog-index, so add the pin yourself, and add --deep for the object-level check (renderVerifySection, console/src/screens/keys/offline-recovery.ts).

   There is also a keyless attestation, POST /admin/restore/attest, that checks the signature, completeness and anti-rollback without a key and without touching data. It is a useful in-account cross-check, but treat it as an engine-side assertion. It is not a Cloudflare-side verdict, and during a presumed compromise the engine and the account are exactly what you have stopped trusting, so the clean-machine downpipe verify against your pinned signer is the verification that counts.

4. Rotate every Cloudflare credential and the destination keys

   This is the closest thing to containment, and it is Owner-gated. In the Cloudflare dashboard, work through the account itself.

   1. Remove or re-verify every account member, and force re-authentication across the account.
   2. Rotate every API token. A standing deploy token should not exist at all; if one does, that is its own finding to fix. See Cloudflare token scopes for the minimal, short-lived posture.
   3. Revoke Cloudflare Access sessions, then re-check the Access policies and the identity provider behind them.
   4. Rotate the destination credentials at the destination provider. For an S3-compatible destination, rotate DEST_ACCESS_KEY_ID and DEST_SECRET_ACCESS_KEY there, because the attacker read them from the account.

   In the engine, terminate every native session, then kill the bootstrap bearer if it is still live. Revoking Cloudflare Access in step 3 does not touch the engine’s own sessions: the engine accepts a verified Cloudflare Access JWT as its highest-precedence credential but falls through to its own native session cookie (and, unless disabled, the ADMIN_TOKEN bearer), so cutting the Access JWT leaves a passkey, recovery-code, OIDC, or SAML cookie session live. Use the Owner-only Sign out everyone control on the Access screen, in Sessions and passkeys. It deletes the engine’s session signing key so every outstanding native session fails its next check and an attacker riding one is evicted.

   Recovery codes are not affected. They are hashed under their own recovery key, a separate record that is materialised before the session key is deleted, so a printed recovery sheet keeps working across a sign-out-everyone. Do not regenerate the codes as part of this step: regenerating replaces the working set, and the sheet an Owner is holding is exactly what you need if the attacker also took a passkey. Regenerate only when you have reason to believe the codes themselves were captured, and only once you can print and distribute the new sheet. For a single named principal rather than a global sign-out, use the per-user terminate-user control instead.

   Both of those controls, and retiring the bearer below, are step-up gated: each asks for a fresh passkey assertion from the person running the eviction. Have your own authenticator to hand before you start this step, because a missing authenticator stops these controls. The key install, the destination change, the identity-provider connections and the role changes in this runbook are step-up gated too. A bare break-glass token session is exempt, and a Cloudflare Access session passes only while its Access sign-in is less than five minutes old. If you have already cut Access in step 3, you are on a cookie session and will be prompted.

   Then handle the bearer: if you trust the engine enough to drive it, retire the token in-app (Security Centre, Retire break-glass token); otherwise delete the secret with your rotated operator credentials (npx wrangler secret delete ADMIN_TOKEN). A live shared bearer is one human holding two identities, which quietly defeats dual control, so it has no place in a hardened account.

5. Re-stand-up clean, or recover in place

   Now choose a path. Both keep the old archives readable with the old break-glass key; record that in your incident notes either way.

   The cleaner path is to re-stand-up in a fresh account. Deploy the engine and console from a release whose artefact digest you verified against the published checksum. Run a new key ceremony (a new break-glass pair, a new signer, fresh recovery codes, a new printed recovery sheet). Configure a destination you control, bind the sources as they exist in the new account, and resume schedules. Keep the old bucket (or copy its bytes out) and keep the old identity.key and old signer.pub for as long as those archives matter, because old archives open only with the old break-glass key and verify only under the old pinned signer.

   The other path is to recover in place after the rotation above. Redeploy the engine from a verified release into the rotated account, and do not resume schedules yet. The re-key installs a new signer and a new break-glass key, but it does not replace every in-account key. Do the numbered steps below in order, and run the re-key only at step 2.

   Before it writes any new secret, the re-key verifies every destination’s backup RUNLOG under the signer being replaced. A destination whose RUNLOG does not verify there stops the whole re-key, so resolve it with an Owner’s attended re-sign first. Once the re-key proceeds, it hands each verified RUNLOG over to the new signer. It holds new run allocations for up to fifteen minutes while that runs. A destination it could not hand over in time stays pending; the same attended route completes it.

   If the engine held an operational key, that key is burnt too. Tick the “Keep strict custody” box for the re-key. The re-key then sets the break-glass-only marker and deletes OPERATIONAL_PRIVATE and OPERATIONAL_PUBLIC, and the audit log records operational-removed. If you clear the box instead, the re-key writes a new operational pair over the burnt one.

   Do not assume that the re-key removed the burnt key. Before you resume schedules, open the posture score in the security centre. If you ticked the box, confirm that the check operational-retired-key-held (“Break-glass-only choice fully applied”) is in the list and passes.

   If the audit log records key-install-failed, the re-key did not complete. Some of the new keys can be in place, so keep the identity.key from that attempt. Select Install keys to your engine again. The key set stays on the page after the error.

   If the audit log records key-removal-failed, the removal did not complete. The new signer and the new break-glass key are in place, so keep the identity.key from that attempt. Select Install keys to your engine again. The second install writes the same keys and finishes the removal.

   If the check fails or is not in the list, run the break-glass-only switch and check again. The switch can refuse until you confirm that you hold the old identity.key. Confirm only when you hold it.

   Do not assume that the deletion protects the archives sealed while the operational key was in the account. The attacker could copy that private key, and a copy opens every archive sealed to it. The deletion stops the engine using the key. It does not take back that exposure, so treat those archives as readable by the attacker. After a strict re-key the engine cannot open any archive sealed before the re-key. Of the keys you hold, only the old identity.key opens them, and only the old signer.pub verifies them.

   The re-key does not replace a configuration key that the engine already holds (installEngineSecrets, engine/src/admin/attach.ts). The burnt configuration keys open the sealed configuration export and decrypt the credentials that the console stores. If you delete them before the re-key, the re-key installs new ones.

   Before step 1, make sure that you can sign in with a passkey or a recovery code. A sign-in through an OIDC or OAuth2 connection that has a client secret fails until step 3.

   1. In the Cloudflare dashboard, open the engine Worker’s settings and delete the secrets CONFIG_RECIPIENT_PRIVATE, CONFIG_RECIPIENT_PUBLIC and CONFIG_WRAP_KEY.

   2. Run the re-key on the Keys screen. The engine holds no configuration key, so the re-key installs new configuration keys with the new signer and the new break-glass key. It also encrypts with the new wrap key each credential that is stored without encryption.

   3. In the console, enter each stored credential again with its rotated value: the destination secrets, the read-only discovery token, the integration and notification secrets, and any identity provider client secret. To replace a client secret, rotate it at the identity provider first. Then remove the identity provider connection and add it again with the same connection id. The same id keeps the redirect URI and the role of each member. The new wrap key cannot decrypt a credential that the burnt key encrypted, so a run, an integration or a sign-in that needs that credential fails until you enter it again.

   4. Replace each Worker secret that the attacker could read. The console cannot write a Worker secret value, so do this in the Cloudflare dashboard, in the engine Worker’s settings:

      • SCIM_BEARER_TOKEN, if it is set. Put the new value in the SCIM connector of your identity provider too.
      • DEST_ACCESS_KEY_ID and DEST_SECRET_ACCESS_KEY, if you set the destination through Worker secrets. Use the values that you rotated at the destination provider.
      • DISCOVERY_API_TOKEN, if you set it as a Worker secret. Use a new read-only token.
      • BEACON_INGEST_KEY, if the usage beacon is on. Delete it, or ask support for a new key.

      Recording each rotation is optional. It resets the rotation reminder for that secret. An Owner records it with POST /admin/secrets/rotated and the body {"secret": "<id>"}, from a console session or a Cloudflare Access assertion. The id is scim_bearer_token, dest_secret_access_key, discovery_api_token or admin_token. The console has no control for this call. The admin endpoints reference and authentication and authorisation give the details.

   If step 2 fails with “the engine’s own account is not set; add a read-only discovery token or pick the account under Sources before installing keys (no command line needed)”, the engine cannot name its own account. Enter a new read-only discovery token, or pick the engine’s account under Sources. Select Install keys to your engine again.

   From step 1 until you finish step 4, keep the schedules paused. If you do not want to delete Worker secrets yourself, use the fresh-account path above. Its key ceremony makes new configuration keys with the rest of the key set.

   If you deleted the configuration keys only after the re-key, use the Add configuration keys card on the Keys screen to install new ones. Do step 3 and step 4 after that. The card shows when the engine holds a signer and does not hold a configuration key. It makes the new keys in the browser, and the engine installs them with a one-shot token.

   Add configuration keys refuses a new recovery key while CONFIG_RECIPIENT_PUBLIC stays. When only the old CONFIG_RECIPIENT_PRIVATE stays, the engine keeps that private key, writes the public key made from it and does not install the new pair, so the old key stays in use. The card adds only the keys that the engine does not hold, so an old CONFIG_WRAP_KEY that stays also stays in use. Make sure that you deleted all three secrets.

   A configuration recovery key that step 2 or the card installs opens only the configuration exports that the engine seals after that. A configuration export sealed before the re-key opens only offline, with the old identity.key and the downpipe reader’s unseal-export command.

   Then prove the pipeline before you trust it again. With an operational key, run a drill and a scheduled restore test. In the strict posture the engine runs neither, so recover one new run with the new identity.key, as prove recoverability describes.

The signer-presumed-lost branch

The signer private lived in the account, so a full compromise means you should presume it lost. Choose between two branches, and write down which you chose and why.

The defensible default is signer presumed lost. Rotate SIGNER_PRIVATE through the ceremony so future archives sign under a new key. Old archives still verify, because the reader verifies each run against the signer you pin for it: the operator pins both, the new key for new runs and the old signer.pub only for genuinely-old runs captured before the compromise. That last qualifier matters, because a stolen signer can re-sign a tampered run under the old key and it will still pass verify against the old pin, so the old pin is a trust anchor only for runs you held before the account fell. Re-print the recovery sheet with the new fingerprints, retain the old sheet and the old signer.pub, keep the --min-runlog-index pin current, and lean on the out-of-band audit head hash you recorded at the start to bound which runs are genuinely yours. A stolen signer can forge plausible-looking runs that verify until your rotation lands, so treating the signer as burnt and rotating it promptly, backed by the run-log index pin, is what bounds the window.

The old sheet’s fingerprint cannot stand in for the old signer.pub. After the rotation, the next run overwrites the bucket’s copy of signer.pub with the new signer’s key. Only the old file verifies the runs the old signer sealed.

Keeping the signer is only defensible if the evidence shows the secret was never readable, which is rare in a full compromise. If you do keep it, record why in the incident notes rather than leaving it implicit.

Who to page and what they need

A compromise needs the right people awake with the right credentials. Plan this before the incident, not during it.

Fix the topology before an incident, not during one. The Security Centre warns you ahead of time: a two-owners finding when you have a single Owner, a dispose-bootstrap-token finding while a live bearer remains, and a recovery-codes-low finding when your remaining codes run low. They are the same lesson said three ways, because dual control quietly fails when one person can present two identities.

Communication discipline

The audit chain is the timeline of record, and it records denied attempts. Privileged actions during the incident land in it, hash-chained, including the eviction steps themselves: role changes, the bearer retire, credential grants and revokes, restore requests and approvals. A refusal on the owner-action approve and reject route is counted, not chained, and the 401 that opens a step-up records nothing. Run the incident through the product wherever the product can see it. Narrate the rest, such as dashboard actions and provider tickets, in your own notes with timestamps.

At the end, verify and seal the record. Run GET /admin/audit/verify, then export the chain again with GET /admin/audit/export and record its head hash out of band. Attach that closing export beside the evidence you captured at the start, so the post-incident review argues from the tamper-evident record rather than from memory.

Where this fits

For what to expect from a key rotation outside an incident, whether you are rotating the break-glass key, adding an operational key, or re-keying completely, see rotating your keys. For the calmer first-response decision tree, and how to tell a configuration fault from a genuine compromise before you start here, see incident response. For the posture choice that decides whether the engine holds a decryption-capable operational key, and how the break-glass-only posture removes it, see recovery postures. For the key material you rotate and the ceremony that mints fresh keys and a new recovery sheet, see the key ceremony and recovery kit. For the offline verify and restore that keep recovery vendor-independent, see break-glass offline recovery. For the minimal, short-lived token posture that means there is no standing credential for an attacker to find, see Cloudflare token scopes.

Last updated 10 October 2026.