Skip to content
downpipes docs

Troubleshooting

This page is a symptom-to-cause-to-fix lookup for the problems you hit when a backup run fails, when a call to the engine is refused, when a restore does not do what you expected, or when the offline reader surprises you. Each entry names the cause from the code path that produces it and the concrete fix, and where a symptom traces to a capability the console does not yet expose, it says so plainly rather than implying a screen exists. Most of it is written for a developer with a failing call in front of them; the first section is written for whoever opened the console and found a run that failed overnight.

For the full wire-level catalogue of HTTP status codes and named error bodies, the canonical home is error and status codes; this page is the symptom-first companion to it. For the offline reader’s numeric process exit codes, see the CLI exit codes.

A backup run failed

Start with the reason on the row. A failed run always carries one, it is always a member of a closed list, and it is never a stack trace or a raw message, because a raw message can name a record or an object key and the archive encrypts those end to end.

The whole list is below. It is generated from the engine function that produces it (coarseRunError, engine/src/seal/slice.ts), and it is the same table published on runs and history, from the same generated snapshot, so the two cannot drift apart.

There are 21 of them, and they are closed: nothing but a member of this list can reach the row. Of those, 16 carry a remedy the engine itself states. The remaining 5 name the system at fault and stop there, and are marked below rather than given an invented fix.

Reason on the failed runWhat it means, and what to do
destination access errorThe engine names the system at fault and states no action, so no remedy is derivable from it. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line.
destination rejected the write (<code>)Read the S3 code in the brackets: it is the store's own reason, passed through rather than collapsed into a class. It is the actionable half of a destination failure, and unlike a throttle it will not clear on its own.
runlog write contendedRetry. This is compare-and-set contention on the run log, and it clears. A destination fault that will not clear is deliberately kept out of this class, so a run that keeps landing here is contending rather than misconfigured.
shard enumeration incomplete (refused to sign a truncated archive)Nothing was signed, which is the point: the engine found fewer shards than the checkpoint declared and refused to seal a short archive as a complete one. The found and expected counts are not on the run row; collect a support pack, which carries them.
checkpoint unwrap failed (resume state unreadable; signer may have rotated)The in-flight run cannot resume and will not recover by retrying that run. If the signing key was rotated while the run was in flight, that is the cause: the checkpoint was wrapped under the previous signer. Start a fresh run.
seal DO handoff refused (<class>)
<class> is one of invalid-payload, worker-spawn-refused, do-error, unreachable.
The fault is inside the engine's own control plane, not at your destination. The sub-class in the brackets says which handoff refused and is re-gated against a closed list, so it is never a passed-through message. Retry the run; if the same sub-class repeats, raise it with the cause digest.
seal DO handoff refusedAs above, with no sub-class recovered. Read it as the engine's control plane, not your destination: this class exists because these refusals used to arrive as "status 500" and be classified as a destination fault, which blamed the customer's bucket for an engine-side problem.
internal invariant violatedChange nothing about your configuration. The engine states this is its own bug rather than a misconfiguration, so no source, destination or schedule change will clear it. Collect a support pack and raise it.
shard id overflow (refused to seal a misordered archive)The engine names the system at fault and states no action, so no remedy is derivable from it. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line.
resume state divergent (mid-record resume refused)The engine names the system at fault and states no action, so no remedy is derivable from it. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line.
source cannot range-read (chunked capture unavailable)The record is too large to take in one piece and its source offers no ranged read, so it can never be chained across windows. The size ceiling is not the thing to change: capture that record out of band. The engine names this class ahead of the ceiling class for exactly that reason.
record over the in-band size ceiling (recover it out of band)Recover that record out of band, as the class says. This is a documented ceiling with a documented remedy rather than a source fault, and the rest of the run is unaffected.
unsupported source typeThis is a configuration fault, not a source fault: the downpipe names a source type this engine build cannot back up. Do not go looking at the source, which is healthy. Remove or replace the source, or move to an engine build that supports it.
source resource missingThe binding is present but the thing it points at (the KV namespace, R2 bucket or D1 database) is gone. Re-create the resource or repoint the source. This class exists so you are told the resource is missing rather than handed a destination error for a source-side fault.
source binding errorThe binding the downpipe names is not present in this deployment. The usual cause is a deploy that replaced the worker's bindings; deploy with the command that reads the live bindings and ships a superset, and check the Sources screen for a binding flagged missing.
source too large to back up in one slice (not yet resumable)There is no in-product remedy today, and the class says so: this source cannot be split across slices yet. Narrow the source so one slice can finish it.
source read errorThe engine names the system at fault and states no action, so no remedy is derivable from it. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line.
cf-config source needs an accountId (re-save the source with an account id)Re-save the source with an account id, as the class says. This is a source saved before an account id was required; validation now refuses one without it up front, so re-saving is the whole fix.
source listing exceeded the page cap (too many items to enumerate in one run; narrow the source selector)Narrow the source selector, as the class says. The surface holds more items than one run can enumerate. This is a discovery failure with a structural cause, not a transient one, so retrying unchanged will land here again.
engine not fully configuredRequired configuration is absent on the engine. Complete the deployment configuration and re-run; no retry clears it while a required value is unset.
run failedThe engine names the system at fault and states no action, so no remedy is derivable from it. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line.

The order is the classification order and it decides ties: the engine takes the first branch that matches, so a fault whose message carries two recognisable shapes is reported as the higher one.

Beside the reason, the row carries a short one-way digest of the raw message, and the engine’s own log line carries the same digest. That is how a failed run in the console is joined to the log line without either of them carrying the text. Quote the digest when you raise a failed run; it is the only handle support has, and the raw message is deliberately unavailable to both of you.

Three failure shapes do not appear as a failed run at all, and each is a different search than the one above.

Symptom Cause Fix
The run reads ok but far fewer records arrived than you expected A run that skipped records, or found records that vanished mid-run, still seals and still reports ok. The shortfall rides on a warn badge beside the status rather than in the status word Read the badge, not the word. The count is on the badge, on a “Not fully captured” row in the run’s detail drawer, and on the summary band’s latest-run tile. See runs and history.
A run vanished: it is neither in the failed facet nor still in flight The engine gave up on a run that never completed and retired it as abandoned. That state has no facet and the filter parser rejects it, so no combination of facets and no ?status= link reaches it Scan the unfiltered table for the warn tone. Reconcile the summary band’s total against the sum of its three tallies: a total that exceeds the sum is abandoned runs, unaccounted for.
A backup pauses for hours on one destination without failing or finishing The destination or the source answered 503 SlowDown or 429, so the engine parked the run on a patient retry ladder with its checkpoint preserved rather than abandoning a backup that only needs to wait Let it ride. A parked run resumes from its checkpoint with no double work, and a throttle never counts against the hard-fault strike budget. The destination row below covers the tuning.

Sign-in and authentication

Telling the rejection shapes apart is the first diagnostic step, and the trap is that the 401 has two of them. A plaintext unauthorised is a failed sign-in. A JSON body carrying stepUpRequired at the same status is a live session being asked for a fresh identity proof, and it is not a sign-in failure at all. A failed permission check or a cross origin write is a JSON 403.

Status Code When Fix
401 unauthorised (plaintext) No usable credential was presented, or the presented credential did not verify Present a valid credential: a Cloudflare Access assertion, a signed session cookie from a passkey or native single-sign-on login, or the ADMIN_TOKEN bearer.
401 unauthorised (plaintext) A higher-assurance method was present but invalid (a bad Access JWT, or a stale or tampered session cookie) Fix the failing method. The engine never downgrades a present-but-invalid higher method to a weaker one, so a broken cookie does not silently fall through to the token. If a valid token still 401s, check you did not also send a stale Access header or cookie.
401 unauthorised (plaintext) A session verified but carried no usable email or stable subject (for example a Cloudflare Access service token) Use a credential carrying a verified email and subject. Only the bare ADMIN_TOKEN is exempt, because it is the email-less break-glass by design.
401 step-up required (JSON, with stepUpRequired: true) A first-party cookie session called a step-up gated route without a recent enough authentication and without a valid step-up token Run the ceremony at POST /admin/stepup/begin and POST /admin/stepup/finish, then re-send the original request with the returned token in the x-downpipes-stepup header. Do not sign the caller out: the session is valid and nothing was refused on its merits. See step-up re-authentication.
501 passkey_not_configured (JSON) A passkey or step-up ceremony was asked for on an engine with no CONSOLE_ORIGIN set, so there is no origin to bind it to Set CONSOLE_ORIGIN on the engine. Until it is set every step-up gated action fails for every user, and no retry clears it. It arrives alongside the console’s own “Engine CONSOLE_ORIGIN is not set to this console” block error, because the two have one cause.

The anti-downgrade rule is the property to keep in mind: precedence is Access, then the session cookie, then the token, and the first present method that fails stops the request rather than retrying lower (engine/src/admin/router.ts).

A 403 is JSON and comes in three shapes at the auth boundary that need different remedies. Discriminate them by the error field.

Status Code When Fix
403 forbidden (with required and have) The caller authenticated but their role lacks the capability the route needs. required is the missing capability, have is the caller’s role Grant a role that holds the named capability, or have someone who already holds it act. The owner-exclusive capabilities keys.ceremony and posture.riskaccept cannot be conferred to any other role.
403 csrf origin check failed A cookie-borne mutating request (a passkey or native single-sign-on session, anything that is not GET or OPTIONS) arrived with a missing or mismatched Origin Send the request from the configured console origin. The Access and token methods present an explicit header rather than an ambient cookie, so they are never subject to this check.
403 csrf token check failed A cookie-borne request to a session-lifecycle or passkey-management route did not carry an x-downpipes-csrf header matching its __Host-downpipes_csrf cookie. This is a second, independent check on top of the Origin one Call GET /admin/whoami on the cookie session, which returns csrfToken and sets the paired cookie, then echo that value in the header. A client that never called whoami, or a browser that partitioned the cookie, produces this with no cross-site attempt involved.

A forbidden body means change who is calling; a csrf origin check failed body means change where the call comes from; a csrf token check failed body means the call comes from the right place but is not carrying the token that proves it. A client that matches only the first two treats the third as an unexplained denial, which reads as a role problem and is not one. For a custom-role caller the have field reports the viewer floor while the real authority is the caller’s resolved capability set, so read it as “this caller does not hold required”. The full method precedence and the role-by-capability matrix live on authentication and authorisation.

Locked out: the break-glass paths

When single sign-on or a passkey is unavailable, there are two deliberate ways back in, and they are different from the break-glass key that decrypts archives.

Problem Cause Fix
Cannot sign in because Access or the identity provider is down The usual higher-assurance method is unreachable, and no other credential is configured Use the ADMIN_TOKEN bearer fallback: an all-or-nothing owner login that needs no email. It can later be disabled with ADMIN_TOKEN_DISABLED once Access is back. Keep the token handling tight, since it is full owner.
A specific user cannot sign in but the platform is healthy That user’s usual method failed and they hold per-user recovery codes Use the user’s recovery codes to sign in. These are a per-user admin-API path, distinct from the ADMIN_TOKEN and from the break-glass key.

Do not reach for the break-glass key here. The break-glass key is the offline private key that decrypts your archives during recovery, not a console sign-in. The console sign-in fallbacks are the ADMIN_TOKEN and recovery codes, covered on session management; the offline decryption key is covered on break-glass offline recovery.

Restore and recovery

A restore apply has more than a capability gate in front of it, so a 403 on an apply is usually about a missing second approval rather than about your role.

Status Code When Fix
403 restore not approved (with planHash) An apply (POST /admin/restore with confirm: true) passed the restore.apply capability gate but no second identity has approved this exact plan Have a different authorised identity approve at POST /admin/restore/approve with this planHash, then re-submit the apply. A caller cannot approve their own request, and a changed plan produces a new hash that voids any prior approval.
403 forbidden (with required: restore.apply) An operator or viewer attempted an apply An apply needs the restore.apply capability (restore-operator, approver or owner). A dry run, by contrast, writes nothing and is allowed for any authenticated role.
403 prune not approved (with planHash and mode: "not-approved") A retention prune apply reached the same dual-control gate with no approval armed for that plan Have a different authorised identity approve the plan, then re-submit. Match this string separately from restore not approved: they are two routes, and a client matching only the restore one shows the operator a bare denial that reads as a role problem. The body’s retained and superseded run counts say what the refused plan would have done.

The two-step is deliberate: the capability gate runs first so an unauthorised apply is always the forbidden 403 with a denied-apply audit entry, then the dual-control check runs last before any byte is written, and a missing approval is the restore not approved 403 (engine/src/admin/router.ts). The approval is consumed only after a successful apply, so a failed apply leaves it usable for a retry. The full model is on dual control and restore flow.

Recovery capabilities the console does not surface

Some recovery capabilities are real in the engine API but have no console screen in v1. If you go looking for them in the portal, you will not find them, and that is by design rather than a bug.

Problem Cause Fix
A day I expect on the restore calendar is not marked The calendar marks only days holding a SUCCESSFUL run within the bounded fifty-run history ring; a day before the ring’s own floor reads as beyond the shown history (not unrecoverable), and a day whose only runs failed is marked distinctly, never as available Use “Browse all runs by id instead” (the plain run picker) or the Runs screen to reach a run older than the calendar’s shown range; it remains restorable by its id. Call GET /admin/runs/at?downpipe=&at=<rfc3339> directly if you need the raw resolution.
Single-record restore is not on a console screen Granular single-record restore is an engine-API intent (recordName on the restore request); there is no dedicated portal screen for it in v1 Drive it through the restore API with recordName set to the exact record name. An exact-name miss returns an honest “record not found in run”, never a vacuous success. See granular and targeted restores.
A break-glass-only downpipe reports no in-account read-back The engine has no operational private key for that downpipe, so it cannot reopen a sealed run on its own: the scheduled restore test, the automated drill and in-account retention pruning all need a key at a moment when nobody is present to supply one This is the posture working as intended, and it does not put recovery out of reach. Restore from the console’s break-glass panel, supplying your break-glass private in the browser for that restore; it is wiped when the restore finishes and the engine never receives it. The downpipe tool is the alternative when the platform itself is unreachable. Verification at seal and the hourly canary keep running throughout. See recovery postures.

Coverage, assurance and billing

Several surfaces are deliberately honest about what they cannot yet show. Reading their state as broken leads you down the wrong path.

Problem Cause Fix
Coverage shows resources as unknown rather than protected The console cannot populate the resource inventory itself, so with no inventory supplied the coverage view shows the honest-unknown state. It never renders green for a resource it cannot confirm Supply the reference inventory via POST /admin/coverage/inventory so the gap view has a denominator to compare against. Until then, unknown is the correct, honest display, not a fault. See coverage.
There is no live assurance dashboard streaming verification results There is no live assurance dashboard in v1; assurance is proven by running a blind restore test or a keyless attestation, and full verification is out of band Run the recoverability proofs on demand and read their results. Reports assert that a signature is present and well-formed rather than verifying it client-side, because the verifying key is not in the browser. See prove recoverability and reports.
Usage or consumption figures are absent There is no usage producer wired in v1, so there is no live consumption feed to read Treat usage figures as not produced rather than as a broken metric. Cost guidance is a pre-run projection, covered on cost prediction, not a live reading.
You are looking for a billing or subscription-management screen inside the console or control-plane There is none, by design, and there is no payment processor in the product either. Paid plans are quoted and invoiced off-platform, and paid by bank transfer; the only licence-related screen is Licence and updates, where you activate a token. Self-serve Stripe-hosted checkout was withdrawn on 26 July 2026 This is expected, not a missing feature. Every plan, including Starter, Growth, Business and the MSP pack, is now a quote, a purchase order, and an invoice paid by bank transfer, after which a vendor operator mints the licence at the single authenticated mint endpoint for you to activate in the console. There is no Stripe checkout or customer portal to look for any more. See licensing and the control plane.

The update channel

The self-update channel is live: the one-time owner ceremony is complete and the vendor serves a signed channel at the singular update.downpipes.io, so on a base-config deployment configured: true and verified: true is the normal state. A configured: false reading is now the exception, from a deployment that predates the signer being pinned.

Problem Cause Fix
GET /admin/updates reports configured: false The base and demo config pin UPDATE_SIGNER_PUBLIC and set UPDATE_CHANNEL_URL, so a base-config deployment reads configured: true. The generated wrangler.deploy.toml that npm run deploy produces is the committed wrangler.toml copied verbatim plus appended source-binding stanzas, so it never touches the signer or the channel URL; a configured: false reading means this deployment predates the signer being pinned Pin UPDATE_SIGNER_PUBLIC and set UPDATE_CHANNEL_URL in wrangler.toml, then redeploy, after which the engine pulls and verifies the live channel. A configured: false reading is harmless regardless, since updates are never required. See update channel activation.
The deployed Cloudflare version id is not shown in the console The live Cloudflare version id (cfVersionId) is not surfaced in the console in v1, even though the engine self-reports its version metadata at runtime Read the version identity from GET /admin/status, which exposes the deployed version metadata, rather than expecting a console field. The safe-apply self-check uses it internally. See the update channel trust and rollback.
The licence reports tier community with no token set The licence path fails open: a bad, absent or expired licence yields tier community, and LICENCE_SIGNER_PUBLIC is commented out in engine/wrangler.toml until the vendor pins it Activate a licence from the console once the signer is pinned. The licence gates only assurance features, never the data path or recovery, so community blocks nothing. See licensing and the control plane.

cf-config restore is supervised-first

cf-config restore is built and console-wired on main, but it is not deployed to the live demo. The account scope is guarded: an apply into an account that is not provably the archive’s recorded origin is refused unless you type the target account id back. Treat its first live use as supervised rather than fire-and-forget, because re-applying a configuration surface writes to a live Cloudflare account. Of the 313 Cloudflare config surfaces, 60 auto-restore in-band and the remaining 253 are backup-and-preview only, re-applied out of band. See the Cloudflare config backup and restore.

The offline reader (the downpipe CLI)

The reader’s surprises come from honest documentation drift and from sinks that are deliberately bounded. The code is authoritative where a README example disagrees.

Problem Cause Fix
downpipe restore --sink stdout fails with an unknown-sink usage error The valid sinks are file, env and discard; newRestoreTarget in downpipe/cmd/downpipe/restore.go accepts only those three and returns a usage error for anything else Use --sink file with --out <dir> to write one file per record, --sink env to emit dotenv lines to standard output, or --sink discard for a restorability check that verifies every record and writes nothing. There is no stdout sink.
downpipe version prints downpipe dev Neither go install nor a from-source go build runs the release pipeline that would stamp a tag via build flags, so both leave the version at dev. A stamped release binary exists (v0.2.0, signed and SBOM-attested) but sits in a maintainer-reviewed draft release, not yet published A dev version is expected from either install path and does not affect correctness: a dev tool reads and writes the downpipe/0.1.0 archive format unchanged. The tool version and the archive format string are distinct. See the CLI command reference.
restore reports it wrote nothing restore is a dry run by default: it plans the writes and reports conflicts, writing nothing. It also refuses to overwrite existing target state Re-run with --apply to write. The discard sink is the exception: it always runs the full verify path because that is its purpose, regardless of --apply.
A restore exits non-zero after apparently working The reader returns a numeric exit code that states exactly what was proven: not verified, incomplete, a plaintext mismatch, or stale, among others Read the exit code against the CLI exit codes, which is a different contract from the engine’s HTTP statuses. A non-zero exit on a partial restore is the tool refusing to present a partial result as success.

A second drift to know: setup is a dry run by default and creates nothing without --apply (cmdSetup, downpipe/cmd/downpipe/setup.go), so if you expected it to provision on the first invocation, add --apply. It reads your own Cloudflare token from the environment and sends it only to the Cloudflare API; the vendor and the in-account console never see it.

Capture markers in a backup of stream or images

The inventory and binary-capture paths never drop data silently. A symptom that looks like missing data is usually an honest marker you can read.

Problem Cause Fix
A backup of stream or images holds fewer records than expected, with a _truncated record The account inventory exceeded the paging budget, so the adapter emitted an honest _truncated marker at the cap rather than dropping the tail (1000 pages for stream and images) Read the _truncated marker for the cap that was hit. This is a no-silent-gap design, not a fault; the captured set up to the cap is intact and signed.
With binary capture on, a record reads as not captured rather than carrying its bytes Binary capture (includeContent, valid for stream, images and artifacts, though artifacts is not yet a customer-selectable source type) is size-gated: a blob at or below 32 MiB is buffered, and a larger blob whose size the server advertises and that supports range reads is captured by ranged reads up to a ceiling: about 3.66 GiB for a value with a stable ETag that can mid-record-resume across slices, or about 1.84 GiB for a value that cannot resume (no stable ETag, or sliced runs disabled) and so must complete in one slice A larger blob from a server that does not advertise a size or does not support range reads, or one whose declared size exceeds the applicable ceiling, is recorded as an honest not captured marker carrying its size, because a chained capture needs a known length and ranged reads. Treat the marker as the truthful record it is, not a broken capture, and capture the blob out of band if you need its bytes.

Destinations and replication

A destination that looks down is often a single copy of several being unreachable, and the engine is built to keep going.

Problem Cause Fix
A backup reports one destination as down but does not fail In a multiple-destination setup the engine seals to the first reachable destination and reports per-destination state honestly; one copy being unreachable is “N of M copies”, not a failed backup Read GET /admin/replication (optionally ?id=<downpipeId>) for the per-destination “N of M copies” view and which destination is down. A restore falls back across destinations by destinationId, so a lost primary does not block recovery. See failover and replication.
A destination write is denied with an access error The endpoint was reachable and the request was signed, so the access key is valid but is not authorised for this bucket. The common cause is an R2 API token scoped to a different bucket than this destination For a multiple-bucket or three-two-one setup, use an account-wide Object Read and Write token rather than a single-bucket one, and confirm the endpoint’s account id matches the bucket’s account. See the Cloudflare token scopes.
A backup pauses for a long time on one destination without failing or finishing The destination (or the source) returned 503 SlowDown or 429, so the engine parked the run on a separate patient ladder, preserved its checkpoint, and is waiting to resume when the destination recovers rather than abandoning a backup that only needs to wait Let it ride: a parked run resumes from its checkpoint with no double work, and gives up only after a long parking window (tunable with DEST_THROTTLE_MAX_YIELDS). A throttle never counts against the hard-fault strike budget.
Sources silently went missing after a deploy A plain wrangler deploy replaces the worker’s bindings with exactly what the toml lists, and no source bindings live in the toml by design, so a direct deploy drops console-attached sources Always deploy with npm run deploy, which reads the live worker’s bindings and deploys a superset, and which stops rather than ship a binding-dropping config. The Sources screen and the preflight check flag any binding that has gone missing. See binding drift.

Reading an incident decision before you act

When an account is suspected compromised, the order of operations matters more than any single command. The incident runbook (engine/docs/INCIDENT.md) puts the decision tree first: confirm whether the account is presumed hostile before touching it, because eviction steps differ from a routine recovery. If the operational key may be exposed, the break-glass-only posture and an offline recovery with the break-glass key are the path that assumes no in-account trust. The detail is on incident response and account compromise eviction.

Where this fits

These are the symptoms and fixes; the pages below give the surrounding contracts and models.

Last updated .