Skip to content
downpipes docs

Anatomy of a backup run: from cron tick to sealed archive

A backup in downpipes is not a monolithic job. It is a short, budgeted sequence that a scheduled tick starts, that can hand a large workload off to a separate worker object, and that finishes by appending one signed entry to the destination’s run index. Understanding that sequence makes the engine’s behaviour legible: why backups land on roughly a quarter-hour rhythm, how a multi-gigabyte source completes inside a serverless platform’s limits, and why a failure mid-run cannot corrupt your history.

This page is for a developer who wants the conceptual model with just enough architecture to follow it. The operational deep dive (bindings, the deploy hazard, tuning sliced runs) lives in Operations, and the durable-object and platform-limit detail lives in the engine architecture and scale pages.

The components in one breath

A running engine is a small set of cooperating parts.

ComponentWhat it isIts job
downpipe-engine WorkerOne Cloudflare WorkerRuns the cron handler and seals runs; in the default topology it has no public hostname of its own
SchedulerDOA single Durable Object, the authority planeOwns schedules, the monotonic run index, the in-flight lease, run history, roles and the audit log
RunSealDOA per-downpipe Durable ObjectCarries a large run across many invocations as alarm-chained slices
The */15 cronA scheduled triggerDrives every automatic run; it is the sole scheduled dispatch driver, with a console Run now as the on-demand exception
The consoleA separate in-account WorkerReaches the engine over a worker-to-worker service binding, not a URL

The console reaches the engine over a service binding and proxies the admin surface same-origin, so in the default topology the only public surface is the console’s own domain, and the engine’s cron triggers and Durable Objects run without any route (engine/wrangler.toml). A split topology that gives the engine its own custom domain exists for deployments that want separate front doors, but it is not the default.

The run sequence

A single scheduled run moves through a fixed sequence. The order is deliberate: the engine resolves where a run will be written before it allocates one, so a destination that cannot be resolved at all leaves nothing half-started to clean up.

  1. A cron tick fires

    The */15 cron invokes the engine’s scheduled handler, which kicks off the driver in the background (engine/src/index.ts). This tick is what starts runs automatically; the scheduler alarm never does. A console Run now starts a single run the same way on demand, outside the schedule.

  2. Ask the authority plane what is due

    The driver pokes the SchedulerDO to reconcile its timers, then asks it for the set of enabled, not-in-flight downpipes whose next run time has passed. The driver sorts that set oldest-due-first, so a crowded tick serves the most overdue downpipes before it runs low on budget.

  3. Select a destination, before allocating a run

    For each due downpipe the driver resolves the seal destination first. A downpipe that fans out to two or more destinations is probed for the first one that is reachable and writable, by writing and deleting a tiny marker rather than merely listing, so a list-capable but write-denied bucket is caught here (destinationReachable, engine/src/cron/seal-dispatch.ts). This step can end badly in two ways, and they end differently. A destination that cannot be resolved (the pinned destination of a single-destination downpipe whose record was deleted, an unreadable destination config, a malformed CONFIG_WRAP_KEY) throws here, before any run exists, so nothing is allocated and nothing needs cleaning up. A fan-out downpipe skips a destination whose record was deleted and probes the next one. Every destination being unreachable is the other case: the driver still triggers the run, so a run id and a RUNLOG index are allocated and an in-flight row appears, and only then does it post a failed completion reading “all destinations unreachable” and free the lease. That is deliberate, because a backup that silently did not happen is worse than one you can see failed, but it does mean a run index is consumed and the failure is visible in run history rather than absent from it.

  4. Trigger the run

    The driver asks the SchedulerDO to trigger the downpipe. In one storage transaction the DO allocates a ULID run id, allocates the next monotonic run-index, takes the in-flight lease, and appends an in-flight history row (trigger, engine/src/sched/scheduler-do-scheduling.ts). If a previous run is genuinely still in flight, the trigger is coalesced and the new tick does not start a duplicate.

  5. Run the first slice inline

    The engine generates a fresh 256-bit per-run master, then seals the first slice inline on a budget shared across everything this invocation does (sealRunSliced, engine/src/seal/runstate.ts). A small run finishes here, never persists its wrapped master, and behaves like the simple whole-run path.

  6. Finalise inline, or hand off

    If the source is exhausted and there is budget headroom, the engine finalises in the same invocation: it writes the root manifest and signature, the recovery bundle, and appends the run to the index, then reports completion. If not, it hands the run, with its checkpoint and the shards it has flushed, to the per-downpipe RunSealDO.

Why the budget is shared

One cron invocation has a single platform allowance: the per-invocation subrequest cap and the CPU and wall-clock ceiling. Every KV get, every destination put, every fetch to a Durable Object counts against the subrequest cap. So the driver spends one budget across the whole tick: the reconciliation calls, each downpipe’s failover probe and trigger round-trip, and each inline first slice (engine/src/seal/budget.ts). When the budget runs low, it stops starting new downpipes and lets the rest carry to the next tick. The alternative, being killed mid-loop at the platform cap, would persist no progress.

Large runs: alarm-chained slices

A run too large for one invocation continues in the per-downpipe RunSealDO. This is the engine’s answer to a serverless platform’s hard per-invocation limits.

Each alarm wakeup of the seal object gets a fresh subrequest budget. On each wakeup it heartbeats the scheduler’s in-flight lease so a long run is not reclaimed as crashed mid-flight, seals one budget’s worth of records, and persists its advanced checkpoint and that slice’s shards in one atomic put, so the resume cursor can never run ahead of the recorded shards (engine/src/seal/runstate.ts). When the crawl is finally done it finalises (root manifest, recovery bundle, the run-index append) and reports completion.

Because the alarms are durable, a slice that crashes resumes from its checkpoint rather than restarting the whole run, and content addressing makes the re-seal idempotent: a segment already written is recognised and skipped. A run that genuinely cannot make progress is bounded by a maximum slice-failure count and then resolved as failed, so a downpipe is never wedged forever.

The seal never runs inside the authority plane

By design, the heavy work of crawling, encrypting and writing happens in the Worker invocation or in the RunSealDO, never inside the SchedulerDO. The scheduler is the cheap, serial authority plane for schedules, the run index, the lease and the audit log; keeping the seal out of it is what lets it stay fast and contention-free.

Timing: about fifteen minutes

This is the most important thing to internalise, because it shapes what downpipes can promise about recency.

The */15 cron is the sole scheduled dispatch driver. The SchedulerDO keeps durable alarms. An alarm prunes expired records and re-arms the next wakeup time, and when the cron goes silent it also runs the cron’s alert sweep. It does not dispatch a backup (alarm and rearmAlarm, engine/src/sched/scheduler-do-scheduling.ts). A manual Run now is the on-demand exception: it dispatches a single run immediately, but it is an operator action rather than a schedule. Scheduling precision is bounded by the cron interval.

The practical consequence is that the effective minimum interval between backups of a downpipe is about fifteen minutes, regardless of the cadence you set. The console floors the cadence picker at Hourly, so the interface never offers a sub-tick frequency. The engine itself accepts any interval of at least sixty seconds, and one shorter than the tick dispatches at the next tick rather than sooner. Treat downpipes as a backup engine on a quarter-hour rhythm, not a near-real-time replicator. Set recovery expectations from the newest good run rather than from an instant in time.

Reliability invariants

Several invariants keep a run correct under the failures a distributed system actually sees.

A 30-minute in-flight lease bounds how long a triggered run may hold its slot. A run still in flight past that lease is treated as crashed and reclaimed on the next trigger, which marks the orphaned history row abandoned and starts a fresh run; a long sliced run heartbeats the lease so a genuine seal is never reclaimed (leased and heartbeat, engine/src/sched/scheduler-do-scheduling.ts).

Completion is idempotent and keyed on the monotonic run-index. The index is allocated once per run, never reused, and gap-tolerant, so it names exactly one run for all time. A retried or late completion finds an already-resolved row, owns nothing, and is a safe no-op rather than overwriting a newer run’s state.

Run-coalescing means a downpipe whose previous run is still genuinely in flight is not triggered again, so two runs never race for the same downpipe. And the trailing passes that run after the seal loop, the notification, scheduled restore test, replication and digest passes, are all fail-open: a fault in any of them degrades to nothing happening that tick, never a crashed cron and never anything touching the data or recovery path.

Retention pruning is dry-run unless enforced

Deleting old archives is a separate, deliberate action, not something a run does automatically. Retention pruning runs in dry-run by default and reports what it would remove; it deletes nothing unless deletion is explicitly enforced. When it does delete, it first marks the run superseded in the index and never removes a referenced segment ahead of its index entry. See retention and pruning.

The run index is the spine

Every finalised run appends one entry to the run index of the destination it sealed to (_RECOVERY/RUNLOG). This index is a signed, append-only log written with optimistic conditional writes (appendRunlog, engine/src/seal/pipeline.ts). A concurrent run that wins the write race returns a precondition failure, so the loser re-reads and retries rather than dropping an entry, and the append is idempotent on the run id and index.

Under destination failover there is a subtlety worth knowing. A run may seal to a destination that missed the immediately prior account-global run, so each destination keeps its own locally-continuous chain: at the moment of the winning write, the engine recomputes this downpipe’s entire local chain into strict index order, atomically, inside the same conditional write. The signed root manifest carries the same destination-local link, read from the destination the run sealed to. The effect is that failover never forges a forked or dangling chain, and a reader’s rollback check sees a clean, linear local chain wherever a run actually landed.

Where this fits

This page is the conceptual map. The detail and the doing live nearby.

Last updated .