Troubleshooting
This page is a symptom-to-cause-to-fix lookup for the problems you hit when a backup run fails, when a call to the engine is refused, when a restore does not do what you expected, or when the offline reader surprises you. Each entry names the cause and the concrete fix. Where a symptom traces to a capability the console does not expose, the entry says so rather than implying a screen exists. Most of it is written for a developer with a failing call in front of them; the first section is written for whoever opened the console and found a run that failed overnight.
For the full wire-level catalogue of HTTP status codes and named error bodies, see error and status codes; this page is the symptom-first companion to it. For the offline reader’s numeric process exit codes, see the CLI exit codes.
A backup run failed
Start with the reason on the row. A failed run always carries one, it is always a member of a closed list, and it is never a stack trace or a raw message, because a raw message can name a record or an object key and the archive encrypts those end to end.
The whole list is below. The engine produces these reasons in coarseRunError (engine/src/seal/slice.ts), and runs and history shows the same table.
There are 21 of them, and they are closed: nothing but a member of this list can reach the row. Of those, 16 carry a remedy the engine itself states. The remaining 5 name the system at fault and state no action, and their rows say so.
| Reason on the failed run | What it means, and what to do |
|---|---|
destination access error | The engine names the system at fault and states no action. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line. |
destination rejected the write (<code>) | Read the S3 code in the brackets: it is the store's own reason, passed through rather than collapsed into a class. It is the actionable half of a destination failure, and unlike a throttle it will not clear on its own. |
runlog write contended | Retry. This is compare-and-set contention on the run log, and it clears. A destination fault that will not clear is deliberately kept out of this class, so a run that keeps landing here is contending rather than misconfigured. |
shard enumeration incomplete (refused to sign a truncated archive) | Nothing was signed, which is the point: the engine found fewer shards than the checkpoint declared and refused to seal a short archive as a complete one. The found and expected counts are not on the run row; collect a support pack, which carries them. |
checkpoint unwrap failed (resume state unreadable; signer may have rotated) | The in-flight run cannot resume and will not recover by retrying that run. If the signing key was rotated while the run was in flight, that is the cause: the checkpoint was wrapped under the previous signer. Start a fresh run. |
seal DO handoff refused (<class>)<class> is one of invalid-payload, worker-spawn-refused, do-error, unreachable. | The fault is inside the engine's own control plane, not at your destination. The sub-class in the brackets says which handoff refused and is re-gated against a closed list, so it is never a passed-through message. Retry the run; if the same sub-class repeats, raise it with the cause digest. |
seal DO handoff refused | As above, with no sub-class recovered. Read it as the engine's control plane, not your destination: the class keeps an engine-side refusal from being read as a fault in your bucket. |
internal invariant violated | Change nothing about your configuration. The engine states this is its own bug rather than a misconfiguration, so no source, destination or schedule change will clear it. Collect a support pack and raise it. |
shard id overflow (refused to seal a misordered archive) | The engine names the system at fault and states no action. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line. |
resume state divergent (mid-record resume refused) | The engine names the system at fault and states no action. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line. |
source cannot range-read (chunked capture unavailable) | The record is too large to take in one piece and its source offers no ranged read, so it can never be chained across windows. The size ceiling is not the thing to change: capture that record out of band. The engine names this class ahead of the ceiling class for exactly that reason. |
record over the in-band size ceiling (recover it out of band) | Recover that record out of band, as the class says. This is a documented ceiling with a documented remedy rather than a source fault, and the rest of the run is unaffected. |
unsupported source type | This is a configuration fault, not a source fault: the downpipe names a source type this engine build cannot back up. Do not go looking at the source, which is healthy. Remove or replace the source, or move to an engine build that supports it. |
source resource missing | The binding is present but the thing it points at (the KV namespace, R2 bucket or D1 database) is gone. Re-create the resource or repoint the source. This class exists so you are told the resource is missing rather than handed a destination error for a source-side fault. |
source binding error | The binding the downpipe names is not present in this deployment. The usual cause is a deploy that replaced the worker's bindings; deploy with the command that reads the live bindings and ships a superset, and check the Sources screen for a binding flagged missing. |
source too large to back up in one slice (not yet resumable) | There is no in-product remedy today, and the class says so: this source cannot be split across slices yet. Narrow the source so one slice can finish it. |
source read error | The engine names the system at fault and states no action. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line. |
cf-config source needs an accountId (re-save the source with an account id) | Re-save the source with an account id, as the class says. Validation refuses a cf-config source without an account id, so re-saving the source with one is the whole fix. |
source listing exceeded the page cap (too many items to enumerate in one run; narrow the source selector) | Narrow the source selector, as the class says. The surface holds more items than one run can enumerate. This is a discovery failure with a structural cause, not a transient one, so retrying unchanged will land here again. |
engine not fully configured | Required configuration is absent on the engine. Complete the deployment configuration and re-run; no retry clears it while a required value is unset. |
run failed | The engine names the system at fault and states no action. Treat the class as an attribution rather than an instruction: collect a support pack, which carries the correlation digest that joins this row to the engine log line. |
The order is the classification order and it decides ties: the engine takes the first branch that matches, so a fault whose message carries two recognisable shapes is reported as the higher one.
Beside the reason, the row carries a short one-way digest of the raw message, and the engine’s own log line carries the same digest. That is how a failed run in the console is joined to the log line without either of them carrying the text. Quote the digest when you raise a failed run; it is the only handle support has, and the raw message is deliberately unavailable to both of you.
Three failure shapes do not appear as a failed run at all, and each is a different search than the one above.
| Symptom | Cause | Fix |
|---|---|---|
The run reads ok but far fewer records arrived than you expected | A run that skipped records, or found records that vanished mid-run, still seals and still reports ok. The shortfall rides on a warn badge beside the status rather than in the status word | Read the badge, not the word. The count is on the badge, on a “Not fully captured” row in the run’s detail drawer, and on the summary band’s latest-run tile. See runs and history. |
| A run vanished: it is neither in the failed facet nor still in flight | The engine gave up on a run that never completed and retired it as abandoned, which is a separate state from failed | Select the abandoned facet, or open /runs?status=abandoned. The summary band’s Failed or abandoned tile counts both. |
| A backup pauses for hours on one destination without failing or finishing | The destination or the source answered 503 SlowDown or 429, so the engine parked the run on a patient retry ladder with its checkpoint preserved rather than abandoning a backup that only needs to wait | Let it ride. A parked run resumes from its checkpoint with no double work, and a throttle never counts against the hard-fault strike budget. The destination row below covers the tuning. |
Sign-in and authentication
Telling the rejection shapes apart is the first diagnostic step, and the trap is that the 401 has two of them. A plaintext unauthorised is a failed sign-in. A JSON body carrying stepUpRequired at the same status is a live session asked for a fresh identity proof, not a sign-in failure at all. A failed permission check or a cross origin write is a JSON 403.
| Status | Code | When | Fix |
|---|---|---|---|
401 | unauthorised (plaintext) | No usable credential was presented, or the presented credential did not verify | Present a valid credential: a Cloudflare Access assertion, a signed session cookie from a passkey or native single-sign-on login, or the ADMIN_TOKEN bearer. |
401 | unauthorised (plaintext) | A higher-assurance method was present but invalid (a bad Access JWT, or a stale or tampered session cookie) | Fix the failing method. The engine never downgrades a present-but-invalid higher method to a weaker one, so a broken cookie does not silently fall through to the token. If a valid token still 401s, check you did not also send a stale Access header or cookie. |
401 | unauthorised (plaintext) | A session verified but carried no usable email or stable subject (for example a Cloudflare Access service token) | Use a credential carrying a verified email and subject. Only the bare ADMIN_TOKEN is exempt, because it is the email-less break-glass by design. |
401 | step-up required (JSON, with stepUpRequired: true) | A first-party cookie session called a step-up gated route without a recent enough authentication and without a valid step-up token | Run the ceremony at POST /admin/stepup/begin and POST /admin/stepup/finish, then re-send the original request with the returned token in the x-downpipes-stepup header. Do not sign the caller out: the session is valid and nothing was refused on its merits. See step-up re-authentication. |
501 | passkey_not_configured (JSON) | A passkey or step-up ceremony was asked for on an engine with no CONSOLE_ORIGIN set, so there is no origin to bind it to | Set CONSOLE_ORIGIN on the engine. Until it is set every step-up gated action fails for every user, and no retry clears it. It arrives alongside the console’s own “Engine CONSOLE_ORIGIN is not set to this console” block error, because the two have one cause. |
The anti-downgrade rule is the property to keep in mind: precedence is Access, then the session cookie, then the token, and the first present method that fails stops the request rather than retrying lower (engine/src/admin/router.ts).
A 403 is JSON and comes in three shapes at the auth boundary that need different remedies. Discriminate them by the error field.
| Status | Code | When | Fix |
|---|---|---|---|
403 | forbidden (with required and have) | The caller authenticated but their role lacks the capability the route needs. required is the missing capability, have is the caller’s role | Grant a role that holds the named capability, or have someone who already holds it act. The owner-exclusive capabilities keys.ceremony and posture.riskaccept cannot be conferred to any other role. |
403 | csrf origin check failed | A cookie-borne mutating request (a passkey or native single-sign-on session, anything that is not GET or OPTIONS) arrived with a missing or mismatched Origin | Send the request from the configured console origin. The Access and token methods present an explicit header rather than an ambient cookie, so they are never subject to this check. |
403 | csrf token check failed | A cookie-borne request to a session-lifecycle or passkey-management route did not carry an x-downpipes-csrf header matching its __Host-downpipes_csrf cookie. This is a second, independent check on top of the Origin one | Call GET /admin/whoami on the cookie session, which returns csrfToken and sets the paired cookie, then echo that value in the header. A client that never called whoami, or a browser that partitioned the cookie, produces this with no cross-site attempt involved. |
A forbidden body means change who is calling; a csrf origin check failed body means change where the call comes from; a csrf token check failed body means the call comes from the right place but is not carrying the token that proves it. A client that matches only the first two treats the third as an unexplained denial, which reads as a role problem and is not one. For a custom-role caller the have field reports the viewer floor while the real authority is the caller’s resolved capability set, so read it as “this caller does not hold required”. The full method precedence and the role-by-capability matrix live on authentication and authorisation.
Locked out: the break-glass paths
When single sign-on or a passkey is unavailable, there are two deliberate ways back in, and they are different from the break-glass key that decrypts archives.
| Problem | Cause | Fix |
|---|---|---|
| Cannot sign in because Access or the identity provider is down | The usual higher-assurance method is unreachable, and no other credential is configured | Use the ADMIN_TOKEN bearer fallback: an all-or-nothing owner login that needs no email. It can later be disabled with ADMIN_TOKEN_DISABLED once Access is back. Keep the token handling tight, since it is full owner. |
| A specific user cannot sign in but the platform is healthy | That user’s usual method failed and they hold per-user recovery codes | Use the user’s recovery codes to sign in. These are a per-user admin-API path, distinct from the ADMIN_TOKEN and from the break-glass key. |
Do not reach for the break-glass key here. The break-glass key is the offline private key that decrypts your archives during recovery, not a console sign-in. The console sign-in fallbacks are the ADMIN_TOKEN and recovery codes, covered on session management; the offline decryption key is covered on break-glass offline recovery.
Restore and recovery
A restore apply can have more than a capability gate in front of it. Whether it does is your choice: requiring a second approver is optional and off by default. On an estate that has not turned it on, a 403 on an apply is about your role or your session, not about an approval, and restore not approved will not appear at all. On an estate that has, it is usually about the approval rather than the role.
| Status | Code | When | Fix |
|---|---|---|---|
403 | restore not approved (with planHash) | The estate requires a second approver, and an apply (POST /admin/restore with confirm: true) passed the restore.apply capability gate but no second identity has approved this exact plan. It cannot occur on an estate that has not turned the requirement on | Have a different authorised identity approve at POST /admin/restore/approve with this planHash, then re-submit the apply. A caller cannot approve their own request, and a changed plan produces a new hash that voids any prior approval. |
403 | forbidden (with required: restore.apply) | An operator or viewer attempted an apply | An apply needs the restore.apply capability (restore-operator, approver or owner). A dry run, by contrast, writes nothing and is allowed for any authenticated role. |
403 | prune not approved (with planHash and mode: "not-approved") | A retention prune apply reached its dual-control gate with no approval armed for that plan. Note that prune approval is NOT the restore-approval policy: this gate is not the one you choose at setup, so a prune not approved can occur on an estate where a restore needs no second approver | Have a different authorised identity approve the plan, then re-submit. Match this string separately from restore not approved: they are two routes, and a client matching only the restore one shows the operator a bare denial that reads as a role problem. The body’s retained and superseded run counts say what the refused plan would have done. |
The two-step is deliberate: the capability gate runs first so an unauthorised apply is always the forbidden 403 with a denied-apply audit entry, then the dual-control check runs last before any byte is written, and a missing approval is the restore not approved 403 (engine/src/admin/router.ts). The approval is consumed only after a successful apply, so a failed apply leaves it usable for a retry. The full model is on dual control and restore flow.
From engine 0.3.6 a retention prune apply that did not delete everything it planned answers mode: "partial" with the removed, hidden and refused counts. A refusal counts as a lock only when the object’s lock headers show one, or when Azure’s refusal code names one. A refusal with no lock is usually a missing delete permission on the destination credential. When the partial apply superseded any run, a retry needs a fresh request and approval, because the next plan’s hash differs.
Recovery behaviour that looks like a missing screen
Each row below is recovery behaviour that can look like a missing or broken console feature. The fix column says where it lives.
| Problem | Cause | Fix |
|---|---|---|
| A day I expect on the restore calendar is not marked | The calendar marks only days holding a SUCCESSFUL run within the bounded fifty-run history ring; a day before the ring’s own floor reads as beyond the shown history (not unrecoverable), and a day whose only runs failed is marked distinctly, never as available | A run older than the calendar’s shown range is restorable by its id from your default destination while your retention keeps it. Type that id in the Run id field on the restore form. Call GET /admin/runs/at?downpipe=&at=<rfc3339> directly if you need the raw resolution. |
| Restoring one record | Single-record restore is an exact-name match (recordName), not a prefix | On the restore plan form, enter the exact record name in Restore one record (optional), or use the Restore just this action on a row of a built plan, or send recordName on the restore API. An exact-name miss returns “record not found in run”, never an empty success. See granular and targeted restores. |
| A break-glass-only downpipe reports no in-account read-back | The engine has no operational private key for that downpipe, so it cannot reopen a sealed run on its own: the scheduled restore test, the automated drill and in-account retention pruning all need a key at a moment when nobody is present to supply one | This is the posture working as intended, and it does not put recovery out of reach. Restore from the console’s break-glass panel, which takes your break-glass key in the browser. Only that run’s key goes to the engine. Your break-glass key never leaves the browser, and the browser clears both keys when you leave the panel. The downpipe tool is the alternative when the platform itself is unreachable. Verification at seal and the hourly canary keep running throughout. See recovery postures. |
Coverage, assurance and billing
Several surfaces deliberately show a gap or an unknown instead of a value. Reading their state as broken leads you down the wrong path.
| Problem | Cause | Fix |
|---|---|---|
| Coverage shows resources as unknown rather than protected | The console cannot populate the resource inventory itself, so with no inventory supplied the coverage view shows unknown. It never renders green for a resource it cannot confirm | Supply the reference inventory via POST /admin/coverage/inventory so the gap view has a denominator to compare against. Until then, unknown is the expected display, not a fault. See coverage. |
| There is no live assurance dashboard streaming verification results | There is no live assurance dashboard; assurance is proven by running a blind restore test or a keyless attestation, and full verification is out of band | Run the recoverability proofs on demand and read their results. Reports assert that a signature is present and well-formed rather than verifying it client-side, because the verifying key is not in the browser. From reader 0.3.4, downpipe verify-report checks a report’s signature offline against your signer.pub. See prove recoverability and reports. |
| Usage or consumption figures are absent | No usage producer is wired, so there is no live consumption feed to read | Treat usage figures as not produced rather than as a broken metric. Cost guidance is a pre-run projection, covered on cost prediction, not a live reading. |
| You are looking for a billing or subscription-management screen inside the console or control-plane | There is none, and there never will be. A self-serve Business subscription is managed entirely through Stripe’s own hosted checkout and customer portal, never a downpipes screen, and the renewal notice we email about 30 days before each renewal links the portal’s sign-in page; Enterprise, Custom and MSP / MSSP are quoted and invoiced off-platform, paid by bank transfer, with no checkout of any kind. The only licence-related screen either way is Licence and updates, where you activate a token | This is expected, not a missing feature. For Business, checkout, updating your card, cancelling and changing plan are all Stripe-hosted pages, by design: no card data ever reaches this product. For Enterprise, Custom or MSP, a vendor operator mints the licence at the single authenticated mint endpoint after the invoice is paid, for you to activate in the console. See licensing and the control plane. |
The update channel
The vendor serves a signed channel at update.downpipes.io, and the base config pins its signer and sets its URL, so on a base-config deployment configured: true and verified: true is the normal state.
| Problem | Cause | Fix |
|---|---|---|
GET /admin/updates reports configured: false | The shipped base config pins UPDATE_SIGNER_PUBLIC and sets UPDATE_CHANNEL_URL, so a base-config deployment reads configured: true. The generated wrangler.deploy.toml that npm run deploy produces is the committed wrangler.toml copied verbatim plus appended source-binding stanzas, so it never touches the signer or the channel URL; a configured: false reading means this deployment’s wrangler.toml has lost or overridden UPDATE_SIGNER_PUBLIC or UPDATE_CHANNEL_URL | Restore UPDATE_SIGNER_PUBLIC and UPDATE_CHANNEL_URL in wrangler.toml to the shipped values, then redeploy, after which the engine pulls and verifies the live channel. A configured: false reading is harmless regardless, since updates are never required. See update channel activation. |
| The deployed Cloudflare version id is not shown in the console | The live Cloudflare version id (cfVersionId) is not surfaced in the console, even though the engine self-reports its version metadata at runtime | Read the version identity from GET /admin/status, which exposes the deployed version metadata, rather than expecting a console field. The update self-check uses it internally. See the update channel trust and rollback. |
The licence reports tier community with no token set | The licence path fails open: a bad, absent or expired licence yields tier community. The vendor licence signer is built into the engine (DEFAULT_LICENCE_SIGNER_PUBLIC, engine/src/licence-pins.ts), and the commented-out LICENCE_SIGNER_PUBLIC in engine/wrangler.toml is only an optional override | Activate a licence from the console. The licence gates no product feature, and never the data path or recovery, so community blocks nothing. See licensing and the control plane. |
An in-band config restore writes to the account you name
The account scope is guarded: an apply into an account that is not provably the archive’s recorded origin is refused unless you type the target account id back. Treat its first live use as supervised rather than fire-and-forget, because re-applying a configuration surface writes to a live Cloudflare account. Of the 313 Cloudflare config surfaces, 60 auto-restore in-band, 228 are backup-and-preview only and re-applied out of band, and 25 carry an off-by-default writer a restore reaches only by naming the surface explicitly. See the Cloudflare config backup and restore.
The offline reader (the downpipe CLI)
The reader’s surprises come from safe defaults and from sinks that are deliberately bounded.
| Problem | Cause | Fix |
|---|---|---|
downpipe restore --sink stdout fails with an unknown-sink usage error | The valid sinks are file, env and discard; newRestoreTarget in downpipe/cmd/downpipe/restore.go accepts only those three and returns a usage error for anything else | Use --sink file with --out <dir> to write one file per record, --sink env to emit dotenv lines to standard output, or --sink discard for a restorability check that verifies every record and writes nothing. There is no stdout sink. |
downpipe version prints downpipe dev | A from-source go build does not run the release pipeline that would stamp a tag via build flags, so it leaves the version at dev. The release pipeline stamps the tag through a linker flag, so a version string appears only on a binary that pipeline built | A dev version is expected from a local build and does not affect correctness: a dev tool reads and writes the downpipe/0.1.0 archive format unchanged. The tool version and the archive format string are distinct. See the CLI command reference. |
restore reports it wrote nothing | restore is a dry run by default: it plans the writes and reports conflicts, writing nothing. It also refuses to overwrite existing target state | Re-run with --apply to write. The discard sink is the exception: it always runs the full verify path because that is its purpose, regardless of --apply. |
verify or restore with --signer-fingerprint exits 2 and says it is refusing the signer public key | From engine 0.3.6 the bucket holds one copy of the signer public key, and every run overwrites it with the current signer’s key. Its fingerprint is not the one you passed. Either someone altered the copy, or you re-keyed after that sheet was printed | Pass --signer with the signer.pub file that matches your sheet. A later sheet’s fingerprint helps only for runs sealed after the re-key |
--signer-fingerprint exits 6 and says the destination has no signer.pub | No engine 0.3.6 or later has written to this bucket, so it holds no copy of the signer public key | Pass --signer with signer.pub from your recovery kit |
downpipe verify-report exits 6 and says the input is the report PDF | From reader 0.3.4, verify-report checks the report JSON. The PDF carries no signature that the tool can check | Give --report the report JSON: the console’s View JSON saved to a file, or the body of GET /admin/reports/<kind>. For the evidence pack, fetch GET /admin/reports/evidence-pack?framework=<id>. Each request generates a new report, so that JSON is not the report that the PDF shows |
| A restore exits non-zero after apparently working | The reader returns a numeric exit code that states exactly what was proven: not verified, incomplete, a plaintext mismatch, or stale, among others | Read the exit code against the CLI exit codes, which is a different contract from the engine’s HTTP statuses. A non-zero exit on a partial restore is the tool refusing to present a partial result as success. |
setup is a dry run by default and creates nothing without --apply (cmdSetup, downpipe/cmd/downpipe/setup.go), so if you expected it to provision on the first invocation, add --apply. It reads your own Cloudflare token from the environment and sends it only to the Cloudflare API; the vendor and the in-account console never see it.
Capture markers in a backup of stream or images
The inventory and binary-capture paths never drop data silently. A symptom that looks like missing data is usually a marker you can read.
| Problem | Cause | Fix |
|---|---|---|
A backup of stream or images holds fewer records than expected, with a _truncated record | The account inventory exceeded the paging budget, so the adapter emitted a _truncated marker at the cap rather than dropping the tail (1000 pages for stream and images) | Read the _truncated marker for the cap that was hit. This is a no-silent-gap design, not a fault; the captured set up to the cap is intact and signed. |
With binary capture on, a record reads as not captured rather than carrying its bytes | Binary capture (includeContent, valid for stream, images and artifacts, though artifacts is not a customer-selectable source type) is size-gated: a blob at or below 32 MiB is buffered, and a larger blob whose size the server advertises and that supports range reads is captured by ranged reads up to a ceiling: about 3.66 GiB for a value with a stable ETag that can mid-record-resume across slices, or about 1.84 GiB for a value that cannot resume (no stable ETag, or sliced runs disabled) and so must complete in one slice | A larger blob from a server that does not advertise a size or does not support range reads, or one whose declared size exceeds the applicable ceiling, is recorded as a not captured marker carrying its size, because a chained capture needs a known length and ranged reads. Treat the marker as a record of what happened, not a broken capture, and capture the blob out of band if you need its bytes. |
Destinations and replication
A destination that looks down is often a single copy of several being unreachable, and the engine is built to keep going.
| Problem | Cause | Fix |
|---|---|---|
| A backup reports one destination as down but does not fail | In a multiple-destination setup the engine seals to the first reachable destination and reports per-destination state; one copy being unreachable is “N of M copies”, not a failed backup | Read GET /admin/replication (optionally ?id=<downpipeId>) for the per-destination “N of M copies” view and which destination is down. A restore falls back across destinations by destinationId, so a lost primary does not block recovery. See failover and replication. |
| A destination write is denied with an access error | The endpoint was reachable and the request was signed, so the access key is valid but is not authorised for this bucket. The common cause is an R2 API token scoped to a different bucket than this destination | Re-mint the token scoped to this destination’s own bucket, and confirm the endpoint’s account id matches the bucket’s account. Each destination holds its own credential pair, so a multiple-bucket or three-two-one setup needs one scoped token per bucket rather than one wide token across them. Do not reach for an account-wide Object Read and Write token: the engine issues DeleteObject against a destination credential when it prunes, so an account-wide one carries delete authority over every R2 bucket in the account, including any R2 bucket you back up as a source. See multiple destinations for the exact five permissions, and the Cloudflare token scopes. |
| A backup pauses for a long time on one destination without failing or finishing | The destination (or the source) returned 503 SlowDown or 429, so the engine parked the run on a separate patient ladder, preserved its checkpoint, and is waiting to resume when the destination recovers rather than abandoning a backup that only needs to wait | Let it ride: a parked run resumes from its checkpoint with no double work, and gives up only after a long parking window (tunable with DEST_THROTTLE_MAX_YIELDS). A throttle never counts against the hard-fault strike budget. |
| Sources silently went missing after a deploy | A plain wrangler deploy replaces the worker’s bindings with exactly what the toml lists, and no source bindings live in the toml by design, so a direct deploy drops console-attached sources | Always deploy with npm run deploy, which reads the live worker’s bindings and deploys a superset, and which stops rather than ship a binding-dropping config. The Sources screen and the preflight check flag any binding that has gone missing. See binding drift. |
Reading an incident decision before you act
When an account is suspected compromised, the order of operations matters more than any single command. The incident runbook puts the decision tree first: confirm whether the account is presumed hostile before touching it, because eviction steps differ from a routine recovery. If the operational key may be exposed, the break-glass-only posture and an offline recovery with the break-glass key are the path that assumes no in-account trust. The detail is on incident response and account compromise eviction.
Where this fits
These are the symptoms and fixes; the pages below give the surrounding contracts and models.
Error and status codes
The full wire-level catalogue of HTTP status channels and named JSON error bodies.
Runs and history
How to read a run row and its detail drawer, where the shortfall badge sits, and why an abandoned run is in the total and in none of the tallies.
CLI exit codes
The offline reader’s numeric process exit codes, the separate contract for a drill or recovery script.
Authentication and authorisation
The sign-in precedence, the anti-downgrade rule, and the role-by-capability matrix behind a 403.
Dual control
Why an apply can require a second approval bound to the plan hash, and how maker not equal to checker is enforced.
Last updated .