Upgrading a live deployment and rolling back safely

You perform every upgrade yourself, in your own account, with your own credentials. The vendor cannot deploy, modify or read your deployment, so there is no upstream actor who could push a change at you. This page is the runbook a self-hoster follows to take a live engine from one release to the next without surprises. It sets out the rules for backing a release out if it misbehaves.

Two facts shape the whole procedure. First, nothing applies itself: even with the update channel switched on, all it ever does is tell the engine that a newer version exists. Second, the archives on your destination are not part of an engine upgrade at all. They are an open, versioned format, and an upgrade never rewrites a byte that is already there, so everything already in the bucket stays exactly as the engine that wrote it left it.

What an upgrade can change is the format version the engine writes from then on, and you must not assume it cannot. A reader binary implements a fixed list of format major.minor versions, at any patch, and the current list holds only 0.1. The reader refuses every other version with exit 6 rather than making a best effort (checkFormatVersion in internal/format/verify.go). So a release that moves the format version leaves you with two eras of archive in one bucket. A retained reader that predates the new version opens only the older one. That is a required-stop question, and the intervening release notes are where it is answered; it is one of the reasons the section below insists you read them rather than skim the target’s.

Two checks close this off, and both are cheap. Read formatVersion in a recent run’s root manifest, or the pointer under _RECOVERY/ (for format 0.1.0, _RECOVERY/downpipe/0.1.0/FORMAT.md), before and after the upgrade, and confirm whether it moved. Then run a downpipe verify with your retained reader against an archive written after the upgrade, because a reader that opened last month’s runs is not evidence about this month’s.

Before you start: read every intervening release note

This is a required stop, not advice. Read the release notes for every version between your current release and the target, not just the target’s notes. A release can be marked as a required stop that cannot be skipped. A release can also change the schema epoch that bounds whether you can later roll back. Both of those are things you can only learn from the intervening notes, and both change how the rest of this runbook plays out.

Skipping intervening notes is how an operator walks into an upgrade that needed a stepping-stone version first. Skipping them is also how an operator rolls back across an epoch boundary they did not know had moved. The engine has guards for some of this, covered below, but the notes are where the human decision is made.

The upgrade runbook

This is the operator-CLI path, which works on any engine. The one-click console path is covered at the end, and it produces the same outcome by a different route.

1. Get preflight green first

   Run GET /admin/preflight and confirm it is green; the console’s read-only “Check my engine” view, at /setup/finish/check, lists the checks the engine observed. On the account side, run downpipe preflight --account <id>, which lists the account’s available domains, verifies the chosen one is on an active zone, and reports Secrets Store headroom, the Workers plan and R2. Do not upgrade over a red preflight. If you do, you will not be able to tell afterwards whether the upgrade caused a breakage or merely revealed one that was already there.

2. Export your evidence before any major upgrade

   Export the support bundle, the audit log as JSON with its head hash, and the config history. These exports are small and redaction-safe, and they are what makes “restore the pre-upgrade state” a real remediation rather than a hope. For an upgrade across a schema epoch they are not optional, because the pre-upgrade evidence plus the release notes, not a blind redeploy, is the path back across an epoch.

3. Verify artefact digests against the published checksums

   Pull the release into your deployment pipeline and verify the artefact digests against the published checksums before you deploy anything. The engine can self-report the SHA-384 of its deployable bundle in GET /admin/status once deployed, but the check that matters here is the one you do against the published checksum before promotion, so a corrupted or substituted artefact never goes live.

4. Deploy with npm run deploy: engine, then console

   Run npm run deploy from the engine directory, then deploy the console. Use npm run deploy, never a bare wrangler deploy. The reason is that a bare deploy replaces the worker’s binding set with exactly what wrangler.toml lists, which silently drops every source you attached from the console, because those sources live on the worker and not in the committed config. npm run deploy runs the source-binding reconcile first, reading the live bindings and deploying a superset, and it stops rather than ship a binding-dropping config. The full hazard and the reconcile are in deploy safety and bindings.

   cd engine
   npm run deploy        # reconciles console-attached sources, then deploys
   # then deploy the console

   For a cautious rollout, you can upload the new version with zero traffic first and smoke it before promotion. Run npm run sync-bindings so the preview config carries the live sources, then npx wrangler versions upload -c wrangler.deploy.toml. Note the constraint in the next section: a release containing a Durable Object migration cannot use this gradual path.

5. Verify after, at five minutes and at one hour

   Confirm the engine came up correctly and stayed up. Check GET /admin/health returns 200, and GET /admin/status reports the new engineVersion with ready: true. Confirm the audit log carries the engine-version-change event, so the upgrade is itself in the tamper-evident trail. Confirm the next */15 cron tick lands, which you can see as the preflight cron-tick item staying verified with a recent tick in its evidence, and confirm the next scheduled restore test passes. Check the fast signals at five minutes and the slower ones, the cron tick and the restore test, again at an hour.

The post-upgrade signals, gathered in one place:

Rolling back

Rolling back means putting the previous release live again, either by redeploying it or with wrangler rollback. It is useful, but it is not always safe, and the limits below are platform limits.

A rollback never touches state. Durable Object storage, KV, R2 and D1 keep whatever the newer release wrote, so a rollback reverts the code, not the data. That is exactly why it is safe within its bounds, and exactly why it cannot be relied on to undo a schema change.

Across an epoch boundary, the path back is not a blind redeploy. It is the pre-upgrade evidence you exported plus the release notes, used to rebuild the pre-upgrade state deliberately. This is the reason the evidence export is a required step before a major upgrade.

How this relates to the update channel

The console offers a one-click update as the no-CLI path to the same outcome the runbook above produces by hand. It is built on the same safety order: a release is only ever uploaded or promoted after its bytes verify SHA-384 against a signature-verified channel, a rollback target is recorded first, and the promotion is canary-gated with an automatic rollback when the canary comes back dead on the new version. A verdict that is merely incomplete does not roll back: the settle runs inside the window the code swap itself disturbs, so an unresolved flight keeps the verified new version and leaves the record awaiting confirmation from the hourly canary. A release that needs a Durable Object migration is refused for one-click apply and must be applied manually, for the same reason the gradual path cannot ramp one. A release can also carry the console as a component: the one-click apply then updates the engine first, waits for it to settle, and only then deploys the console, and a console that fails to land is rolled back alone while the engine keeps the new version. The component flow is on applying updates.

Two facts complete the picture of that one-click path.

The engine pulls a signed channel and verifies it against a pinned signer; it never receives a push. The shipped base config pins the signer public key and points at the singular update.downpipes.io, where the vendor serves the signed channel, so on a base-config deployment GET /admin/updates reports configured: true and verifies under the pinned signer, and the console shows an available-update posture rather than asking you to configure the channel. The channel is signed static content, so the engine trusts the pinned signature rather than the host, and a channel that does not verify under the pinned signer is reported as unverified rather than applied. The generated wrangler.deploy.toml used for the gradual zero-traffic rollout is not an exception to any of this: the reconcile builds it as the committed wrangler.toml plus the live source bindings appended, and it never touches [vars], so it carries forward the same pinned signer and the same singular update.downpipes.io host, and a deployment built from it reports configured: true exactly as a base deployment does. None of this gates a backup or a recovery: the licence and the update channel are optional and never sit on the data path.

The one-click path preserves your bindings the same way npm run deploy does. The parts that talk to Cloudflare to upload and promote a version read the target script’s live settings first, re-send every non-secret binding verbatim, list the secret binding types so Cloudflare keeps the secrets and their values, and carry over the runtime compatibility fields, so the binding hazard that makes a bare wrangler deploy dangerous is closed on this path too. A binding type the pipeline cannot prove it can carry forward is a refusal before the upload. The refusal names the type and points you at the wrangler deploy path for that release (classifyBindings and the unknown-type guard in engine/src/admin/cf-deploy.ts). Every opaque Cloudflare failure becomes a message naming what was changed and what was not.

An uploaded version is inert until it is promoted and Cloudflare deploys are atomic, so the worst a bad call can do is a brief availability blip on the engine or console worker, which Cloudflare lets you roll back. The data in your bucket and your offline recovery are independent of the running engine and unharmed. The operator-CLI runbook above stays available for any release the one-click apply refuses, and for a Durable Object migration it is the only path.

Where this fits

This page holds the upgrade runbook and the rollback rules. For the binding hazard that makes npm run deploy mandatory and how to recover a dropped source, see deploy safety and bindings. For the two-worker shape an upgrade deploys into and why only the console needs a route, see topology. For what to do when an upgrade or anything else goes wrong on a live deployment, see incident response.

Last updated 25 September 2026.