Break-glass recovery with the offline open-source CLI
This page is about the second way to get your data back: offline, with the open-source downpipe reader and your own break-glass key, when the in-account engine cannot read your backups back for you. It is for the person who holds the recovery kit and needs to recover when the usual console path is unavailable. It frames when and why you reach for this channel. It points you at the CLI command reference for the commands rather than restating the whole walkthrough here.
Most of the time you never touch this. The normal restore is the console, the engine does the work, and a customer never opens a terminal for it. This channel is the floor underneath that: a recovery path that stays open even if the vendor and Cloudflare both disappear, because everything it needs is the destination bucket bytes, the offline keys from your recovery kit, and a reader you can rebuild from source.
When you need it
The recovery posture, set out in full in choosing your recovery posture, decides which read-back paths are open to you. In the strict break-glass-only posture, the engine holds no operational private key, so it cannot decrypt any archive on its own. In that posture a console restore drill reports that there is no standing in-account read-back key. Without a key, the console offers the keyless attestation, and attended verification as the proof that decrypts.
The console also carries an in-console break-glass restore panel: you supply your break-glass key in the browser, where it is read and never uploaded, so only the per-run key for the run you are recovering crosses to your own engine to write the data back. A split M-of-N key works there too, reassembled in the same browser flow. Keeping your private in the browser is defence in depth, not a custody requirement: your own engine never gains a standing key that could decrypt an archive on its own, which is the very posture a break-glass-only estate chose, and the vendor receives nothing whichever path you use.
There are two situations that bring you to the offline channel.
| Situation | Why the offline channel is the path |
|---|---|
| You run break-glass-only | By design the engine holds no standing key that decrypts an archive on its own. You can supply your break-glass key in the browser to drive an in-console break-glass restore, and this offline channel remains the read-back path that needs neither the vendor nor Cloudflare, so rehearse it on a schedule |
| The engine or console is unavailable | An account compromise, a lost engine, or any time you cannot trust the in-account path, you still recover from the bucket bytes plus your offline keys with nothing else in the loop |
In the two-recipient posture you usually restore in the console, because the engine holds an operational key that can read archives back and prove them recoverable on its own. Even then, an offline drill is the only test that proves rather than assumes the bucket and your offline key are enough by themselves, so it is worth rehearsing regardless of posture. See prove recoverability.
What the offline reader does
The reader is the downpipe Go command line. The same binary verifies a run, attests to it without any key, and restores it. All three run from a local copy of the bucket or straight from the destination the archive was written to, issuing reads only. There are two network backends. Together they reach every destination downpipes writes to: --s3-endpoint for R2, an S3-compatible store and Google Cloud Storage, and --azure-endpoint for an Azure Blob container. The behaviour that matters for a safe recovery is that it never writes by surprise and never launders a bad record as a good one.
An Azure Blob destination has its own flag pair
Azure Blob Storage is the one destination not reached through --s3-endpoint. Azure is a different wire protocol rather than an S3-compatible store behind another host, so it takes its own pair, --azure-endpoint with --azure-container, and its credential comes from AZURE_STORAGE_KEY (a storage account access key) or AZURE_STORAGE_SAS_TOKEN (a shared access signature) in the environment rather than from a flag. Pass both halves of the pair, and do not name an S3 endpoint and an Azure endpoint in the same command; either mistake exits 6 rather than being guessed at. An estate whose only destination is Azure Blob has the same offline channel as any other.
prune --apply does not work against an Azure container. The reader’s Azure backend implements the reads a recovery needs and implements no delete, so an applied prune refuses on the missing capability, and retention on that destination stays the engine’s job. The S3 backend lists no objects, so an applied prune also refuses against an --s3-endpoint destination. Nothing on the recovery path is affected by it.
Keeping a second copy on another store is still worth doing. This is the 3-2-1 pattern an estate wants regardless of which flags the reader takes. The per-provider detail is in destination providers compared.
| Behaviour | What it means for you |
|---|---|
| Restore is a dry run by default | restore plans the writes and reports any conflict with state already present, and writes nothing until you pass --apply. So you read the plan before anything changes |
| It writes to a file or an env sink, not to live Cloudflare | A recovery materialises your records as files on disk, or as dotenv lines, for you to place back deliberately. The reader does not write to live KV or R2, which keeps the offline path simple and auditable |
| Every record is reassembled and hash-verified | On apply it reassembles each record’s value and checks it against its signed plaintext hash as it writes, and it refuses to overwrite an existing destination key |
| A bad record is reported, not hidden | A per-record reassemble, verify or write failure is recorded and the restore continues to the next record, then exits non-zero. A partial restore is never dressed up as a success |
The reader produces files or environment variables rather than touching your live resources. Putting the recovered data back into KV, R2 or a worker is therefore a separate, deliberate step you perform. For the boundary between what an archive can and cannot reconstruct, see what restore can and cannot write back.
Proving integrity with no in-account key
The offline channel also gives you an integrity proof when there is no in-account read-back key to rely on. This complements the engine’s own keyless attestation. Two read-only commands cover it without writing anything back.
The verify command checks a run against your break-glass identity and the signer public key. Read its scope carefully, because this is the one place a clean exit can be read for more than it says. Plain verify checks the signed manifests only: the segment bytes holding your record data are not decrypted, and the summary line says so. To prove the data itself, add --deep, which decrypts and hash-verifies every record’s segment bytes on the full restore path while writing nothing. restore --sink discard does the same work through the restore path and produces a digest with it. Treat a bare verify as a structural check and verify --deep as the recoverability proof.
The attest command needs no identity and materialises no plaintext: it always checks the shard hashes and the structural completeness of the signed root without ever opening the encrypted data. Its signature and freshness checks depend on what you give it: with --signer, the signature is cryptographically verified and, if you also pass --min-runlog-index, so is freshness against a rolled-back RUNLOG; without --signer it is fully keyless, reports the signature as unchecked, and a rollback pin given without a signer is a structural, unauthenticated check rather than a cryptographic one. See the CLI command reference for the guarantee in each mode. Adding --check-bundle to verify (or restore) binds the in-bucket recovery instructions to their signed checksums, so the guidance you are following is itself proven genuine.
A clean verify exits zero, and every classified non-zero exit names one specific verdict to act on. Exit 1 is the unclassified fallback for an I/O or unexpected failure, so a drill script should treat it as “could not run”, not as a verdict about the archive. The normative meaning of every code is on exit codes and verification outcomes, so a scheduled drill script can react to the result rather than read prose.
Getting the reader
There is one archive format line and one reader. The format is downpipe/0.1.x and the reader implements exactly it, at any patch. That is deliberately narrow: a reader implements exactly one major.minor of the archive format and refuses every other version with exit 6 rather than making a best effort, because guessing would derive every key under the wrong labels and report an authentication failure over bytes that are perfectly intact.
Your bucket records which version it holds. It is worth reading before a bad day rather than during one: formatVersion in a run’s root manifest, or the pointer at _RECOVERY/downpipe/0.1.0/FORMAT.md. If what you find is not downpipe/0.1.x, stop and raise it rather than reaching for an override, because no override reads a format a build does not implement.
With that established, the reader itself is the Go module github.com/downpipes-io/downpipe, built with Go 1.26 or newer, from the public repository github.com/downpipes-io/downpipe. go install github.com/downpipes-io/downpipe/cmd/downpipe@latest resolves against the public module proxy, and a plain git clone of the repository also works, with no account or engagement needed. Even so, get your copy of the source in advance rather than during a recovery: this page carries no command that fetches it live, because the one route back to your data that needs nobody’s approval on the day should not depend on the module proxy or GitHub being reachable at that moment.
From a copy you hold, every dependency is vendored, so the build downloads no module, and it makes no network call at all provided the recovery machine’s own Go already satisfies the pinned toolchain version. Keep the source and a Go that meets that pin beside your recovery key. With them you can rebuild the reader even if the vendor and Cloudflare are both gone. See install the tool for that pin and what to hold in advance so this is not a surprise mid-recovery.
The reader is MIT-licensed, and that is a different licence from the rest of the platform. The engine and the console are source-available under the Elastic License 2.0; the downpipe reader is open source under the MIT licence, so keeping a copy, rebuilding it, auditing it, or handing it to a third party who recovers on your behalf needs no permission from anyone. That is deliberate. The one component your recovery cannot proceed without is the one carrying the fewest strings.
Confirm the command set before you rely on the binary
Before a real recovery, run downpipe --help and confirm prune, recombine and unseal-export are all listed. The binary in front of you is the thing to check, rather than what the copy it was built from was expected to carry. That check takes five seconds. A plain from-source go build stamps the version as dev, since it does not run the maintainer’s release pipeline, so the version string will not tell you whether the command set is complete. The build commands are in install the tool.
The tool version is a separate thing from the archive format. The binary stamps its own build version, while the on-disk format carries its own semver identity, downpipe/0.1.0, and the two move independently. The reader accepts exactly the downpipe/0.1.x line, so an unstamped dev build opens the archives your engine writes.
What you need in hand
A break-glass recovery needs the destination bucket bytes (a local copy of the whole bucket tree, or read-only credentials for the destination itself: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for an S3-compatible endpoint, or AZURE_STORAGE_KEY or AZURE_STORAGE_SAS_TOKEN for an Azure Blob container), plus three pieces of information from your recovery kit. The first is identity.key, the break-glass private key, which is what decrypts your archives on this offline path. It must live offline, on an encrypted drive, in a password manager or split across custodians. It is never written onto the printed recovery sheet.
The second is a pinned signer: the key the reader uses to confirm that your writer signed a run. That is signer.pub, a file from the key ceremony. From engine 0.3.6 and reader 0.3.4, it can also be the signer fingerprint on your sheet, passed as --signer-fingerprint.
With the fingerprint, the reader uses the signer.pub copy in the bucket’s recovery bundle only if its fingerprint matches. Every run overwrites that copy, so it is always the current signer’s key. A bucket that no engine 0.3.6 has written to has no copy. A run sealed before a re-key needs the old signer’s file.
The third is the anti-rollback pin: the latest trusted RUNLOG index, which you write on the sheet’s blank anti-rollback line and pass to the reader as --min-runlog-index.
Skipping the first two stops recovery outright; skipping the third does not, which is what makes it easy to leave out, and the whole point of the pin is to catch a bucket that has been rolled back to an older, validly-signed run. The sheet is a public artefact by design: it records the fingerprints of the two key files so you can confirm you hold the right pair, and no key material at all. The signer fingerprint is short enough to copy, and that is what makes it usable as --signer-fingerprint. For how that kit is generated and stored, see the key ceremony and recovery kit.
Where this fits
This page is the recovery-context pointer for the offline channel. For installation, every command, every flag and the step-by-step recovery walkthrough, read the CLI command reference. For the meaning of each exit code a verification produces, read exit codes and verification outcomes. For the posture choice that decides whether this is your only read-back path, read choosing your recovery posture. For the normal in-account journey this sits beneath, read the restore flow.
Last updated .