Skip to content
downpipes docs

Recovery in downpipes: restore, prove recoverability, and break-glass

A backup is only as good as the recovery it makes possible, so recovery is the part of downpipes that proves the rest of it was worth doing. This section is for an auditor or a recovery owner who needs the whole shape of it on one page: how data is actually written back, how a backup is proven recoverable without writing anything, and what happens when there is no in-account key to read with.

Two ideas run through everything here. The first is that restoring data and proving recoverability are different acts. Restoring writes verified plaintext back into your live account and changes it. Proving recoverability reads, decrypts and checks an archive and changes nothing. The second is that the one act that mutates customer data, the restore apply, is the most heavily gated path in the product, because the engine does not roll back what it has written.

Restore is the hero flow

Restore is the path you reach for in an incident, and it is built as a deliberate, reviewable journey rather than a single button. An operator picks a run, reads a dry-run plan that writes nothing, a distinct second person approves that exact plan, the apply runs behind an integrity check, and a receipt records who applied it and who approved it.

Dry-run is the default. With confirm absent or false the engine returns a plan of what an apply would do and writes not one byte. Only confirm: true writes, and even then the engine refuses unless three independent conditions all hold: the caller holds the apply capability, a usable dual-control approval is bound to the exact plan by a distinct approver, and a two-phase integrity verify passes. Any one missing and nothing is written.

The full procedure, step by step from plan to receipt, lives on the restore flow. The approval mechanic, including why the approver is compared on a stable identity rather than an email, is on dual control.

Restore cannot be undone

Once an apply begins writing, there is no rollback. A cancel stops further writes but does not undo records already written. This is the reason restore carries the friction it does, and it is stated again wherever an apply is described.

Three recovery channels

There are now three ways to recover, and they answer different needs.

The normal channel is the in-account console and engine with an operational read-back key present, the fully unattended path: pick a run, review the plan, route it through approval, apply, no terminal involved at any point.

The in-console break-glass channel is for the strict break-glass-only posture, what a new estate has shipped with since 2026-07-28, where the engine holds no operational key. You supply your break-glass key in your own browser, either by uploading identity.key or reassembling an M-of-N split. The browser recovers that one run’s key locally, and you preview and apply from the same no-terminal screen, still under dual control. Your break-glass key never leaves your browser; only that one run’s recovered key crosses to your engine, and only to open it.

The offline channel is the downpipe CLI, for a restore beyond the in-account 200-record ceiling, a redirect or D1 table-subset restore under break-glass posture (the in-console panel only restores to original bindings), or when you want the key to touch no browser at all.

Channel When it applies What it needs
In-account console and engine The normal posture, where the engine can read archives back An operator in the portal; the operational read-back key present
In-console break-glass restore Break-glass posture, restoring to original bindings, no terminal Your break-glass key supplied in your own browser; a second approver for the apply
Offline downpipe CLI Break-glass, beyond the in-console panel’s scope, or with no key ever touching a browser The destination bytes plus the customer-held offline break-glass key

The recovery postures that decide which channels are in force are explained in recovery postures, and the key that makes both break-glass channels possible comes from the key ceremony and recovery kit. For attended, evidence-producing verification of a break-glass-only estate, see attended verification; for the offline CLI path in full, see break-glass offline recovery.

Restoring data versus proving it is recoverable

Restoring data is one verb. Proving a backup is recoverable is four more, none of which write to a real resource and none of which reveal record contents.

Verb What it does What it writes
Drill Stride-samples a real run and reads back a sample of its records, verifying each one Nothing
Blind restore test Decrypts every in-scope record of a real run to a discard sink and verifies each plaintext hash Nothing
Keyless attestation Checks signature, completeness and anti-rollback with no decryption key and no data Nothing
The canary Round-trips a known synthetic corpus through an isolated cell, never a customer archive Only its own isolated cell

The first is the destructive restore covered above. The other four are the assurance verbs. The drill, the blind restore test and the keyless attestation all act on your real archives but read only. The canary is structurally different: it exercises the engine’s own write, seal, read and restore machinery against synthetic data in an isolated namespace, so a green canary proves the machinery works, not that any specific real run of yours is recoverable. Treat the canary as a smoke alarm and the blind test or attestation as the proof of a particular archive.

What each verb proves, what it does not, and when to use which (especially keyless attestation in the break-glass-only posture) is the subject of proving recoverability. The canary has its own page at the canary.

Recovery never consults the licence

This matters for an audit, so it is worth stating plainly. Restore and every assurance verb run at the community tier. The recovery path never reads the licence at all. A licence gates assurance reporting features, the packaging of evidence and the surfaces around it, but it can never gate your ability to get your data back or to prove that you can. An expired, absent or invalid licence resolves to the community tier and changes nothing about recovery. The trust model behind this, where the vendor holds nothing of yours, is described in the no custody trust model.

RPO and RTO, briefly

Two recovery objectives frame the whole section, and they answer different questions.

The recovery point objective (RPO) is about freshness: how much recent data a recovery could lose. downpipes expresses your recovery point as the newest good run, the most recent successful backup the engine can restore from. The console shows you that run and how fresh it is. Restoring from an arbitrary continuous instant is not on offer: you always recover from a run, identified by its id, whether you pick it directly or find it through the restore screen’s calendar.

The recovery time objective (RTO) is about speed: roughly how long a recovery would take. downpipes derives an approximate RTO estimate from the throughput it actually measures during restore tests, scaled to the size of the archive. It is an honest projection with a “based on N drills” caveat, and where there is no drill history yet it reports an honest unknown rather than inventing a number. A point-in-time, restore-as-of-a-timestamp resolver exists in the engine API, and the restore screen’s calendar is built directly on it, marking the days that hold a retained run; the portal restore still always picks a run by id, never an arbitrary timestamp.

Where this fits

Last updated .