Skip to content
downpipes docs

Anatomy of a backup run: from cron tick to sealed archive

A backup in downpipes is not a monolithic job. It is a short, budgeted sequence that a scheduled tick starts, that can hand a large workload off to a separate worker object, and that finishes by appending one signed entry to an account-wide index. Understanding that sequence makes the engine’s behaviour legible: why backups land on roughly a quarter-hour rhythm, how a multi-gigabyte source completes inside a serverless platform’s limits, and why a failure mid-run cannot corrupt your history.

This page is for a developer who wants the conceptual model with just enough architecture to follow it. The operational deep dive (bindings, the deploy hazard, tuning sliced runs) lives in Operations, and the durable-object and platform-limit detail lives in the engine architecture and scale pages.

The components in one breath

A running engine is a small set of cooperating parts.

Component What it is Its job
downpipe-engine Worker One Cloudflare Worker Runs the cron handler and seals runs; in the default topology it has no public hostname of its own
SchedulerDO A single Durable Object, the authority plane Owns schedules, the monotonic run index, the in-flight lease, run history, roles and the audit log
RunSealDO A per-downpipe Durable Object Carries a large run across many invocations as alarm-chained slices
The */15 cron A scheduled trigger Drives every automatic run; it is the sole scheduled dispatch driver, with a console Run now as the on-demand exception
The console A separate in-account Worker Reaches the engine over a worker-to-worker service binding, not a URL

The console reaches the engine over a service binding and proxies the admin surface same-origin, so in the default topology the only public surface is the console’s own domain, and the engine’s cron triggers and Durable Objects run without any route (engine/wrangler.toml). A split topology that gives the engine its own custom domain exists for deployments that want separate front doors, but it is not the default.

The run sequence

A single scheduled run moves through a fixed sequence. The order is deliberate: the engine resolves where a run will be written before it commits to running it, so a destination problem leaves nothing half-started to clean up.

  1. A cron tick fires

    The */15 cron invokes the engine’s scheduled handler, which kicks off the driver in the background (engine/src/index.ts). This tick is what starts runs automatically; the scheduler alarm never does. A console Run now starts a single run the same way on demand, outside the schedule.

  2. Ask the authority plane what is due

    The driver pokes the SchedulerDO to reconcile its timers, then asks it for the set of enabled, not-in-flight downpipes whose next run time has passed. The driver sorts that set oldest-due-first, so a crowded tick serves the most overdue downpipes before it runs low on budget.

  3. Select a destination, before allocating a run

    For each due downpipe the driver resolves the seal destination first. A downpipe that fans out to two or more destinations is probed for the first one that is reachable and writable, by writing and deleting a tiny marker rather than merely listing, so a list-capable but write-denied bucket is caught here (destinationReachable, engine/src/index.ts). If every destination is unreachable the run is recorded as an honest failure and nothing is half-started.

  4. Trigger the run

    The driver asks the SchedulerDO to trigger the downpipe. In one storage transaction the DO allocates a ULID run id, allocates the next monotonic run-index, takes the in-flight lease, and appends an in-flight history row (trigger, engine/src/sched/scheduler-do.ts). If a previous run is genuinely still in flight, the trigger is coalesced and the new tick does not start a duplicate.

  5. Run the first slice inline

    The engine generates a fresh 256-bit per-run master, then seals the first slice inline on a budget shared across everything this invocation does (sealRunSliced, engine/src/seal/runstate.ts). A small run finishes here, never wraps its master, and behaves like the simple whole-run path.

  6. Finalise inline, or hand off

    If the source is exhausted and there is budget headroom, the engine finalises in the same invocation: it writes the root manifest and signature, the recovery bundle, and appends the run to the index, then reports completion. If not, it hands the run, with its checkpoint and the shards it has flushed, to the per-downpipe RunSealDO.

Why the budget is shared

One cron invocation has a single platform allowance: the per-invocation subrequest cap and the CPU and wall-clock ceiling. Every KV get, every destination put, every fetch to a Durable Object counts against the subrequest cap. So the driver spends one budget across the whole tick: the reconciliation calls, each downpipe’s failover probe and trigger round-trip, and each inline first slice (engine/src/seal/budget.ts). When the budget runs low it stops starting new downpipes and lets the rest carry to the next tick, rather than being killed mid-loop at the platform cap, which would persist no progress.

Large runs: alarm-chained slices

A run too large for one invocation continues in the per-downpipe RunSealDO. This is the engine’s answer to a serverless platform’s hard per-invocation limits.

Each alarm wakeup of the seal object gets a fresh subrequest budget. On each wakeup it heartbeats the scheduler’s in-flight lease so a long run is not reclaimed as crashed mid-flight, seals one budget’s worth of records, and persists its advanced checkpoint and that slice’s shards in one atomic put, so the resume cursor can never run ahead of the recorded shards (engine/src/seal/runstate.ts). When the crawl is finally done it finalises (root manifest, recovery bundle, the run-index append) and reports completion.

Because the alarms are durable, a slice that crashes resumes from its checkpoint rather than restarting the whole run, and content addressing makes the re-seal idempotent: a segment already written is recognised and skipped. A run that genuinely cannot make progress is bounded by a maximum slice-failure count and then resolved as failed, so a downpipe is never wedged forever.

The seal never runs inside the authority plane

By design, the heavy work of crawling, encrypting and writing happens in the Worker invocation or in the RunSealDO, never inside the SchedulerDO. The scheduler is the cheap, serial authority plane for schedules, the run index, the lease and the audit log; keeping the seal out of it (design F11) is what lets it stay fast and contention-free.

The timing truth: about fifteen minutes

This is the most important thing to internalise, because it shapes what downpipes can promise about recency.

The */15 cron is the sole scheduled dispatch driver. The SchedulerDO does keep durable alarms, but in this version the alarm only re-arms the next wakeup time; it does not itself dispatch a backup (alarm and rearmAlarm, engine/src/sched/scheduler-do.ts). A manual Run now is the on-demand exception: it dispatches a single run immediately, but it is an operator action rather than a schedule. Precise alarm-driven dispatch is a deferred refinement that would need the engine wired so the alarm callback could seal. Until then, scheduling precision is bounded by the cron interval.

The practical consequence is that the effective minimum interval between backups of a downpipe is about fifteen minutes, regardless of the cadence you set. The console reflects this honestly by flooring the cadence picker at Hourly, so the interface never offers a frequency the engine cannot honour. Treat downpipes as a backup engine on a quarter-hour rhythm, not a near-real-time replicator, and set recovery expectations from the newest good run rather than from an instant in time.

Reliability invariants

Several invariants keep a run correct under the failures a distributed system actually sees.

A 30-minute in-flight lease bounds how long a triggered run may hold its slot. A run still in flight past that lease is treated as crashed and reclaimed on the next trigger, which marks the orphaned history row abandoned and starts a fresh run; a long sliced run heartbeats the lease so a genuine seal is never reclaimed (engine/src/sched/scheduler-do.ts).

Completion is idempotent and keyed on the monotonic run-index. The index is allocated once per run, never reused, and gap-tolerant, so it names exactly one run for all time. A retried or late completion finds an already-resolved row, owns nothing, and is a safe no-op rather than overwriting a newer run’s state.

Run-coalescing means a downpipe whose previous run is still genuinely in flight is not triggered again, so two runs never race for the same downpipe. And the trailing passes that run after the seal loop, the notification, scheduled restore test, replication and digest passes, are all fail-open: a fault in any of them degrades to nothing happening that tick, never a crashed cron and never anything touching the data or recovery path.

Retention pruning is dry-run unless enforced

Deleting old archives is a separate, deliberate action, not something a run does automatically. Retention pruning runs in dry-run by default and reports what it would remove; it deletes nothing unless deletion is explicitly enforced. When it does delete, it first marks the run superseded in the index and never removes a referenced segment ahead of its index entry. See retention and pruning.

The run index is the spine

Every finalised run appends one entry to the account-wide run index, a signed, append-only log written with optimistic conditional writes (appendRunlog, engine/src/seal/pipeline.ts). A concurrent run that wins the write race returns a precondition failure, so the loser re-reads and retries rather than dropping an entry, and the append is idempotent on the run id and index.

Under destination failover there is a subtlety worth knowing. A run may seal to a destination that missed the immediately prior account-global run, so each destination keeps its own locally-continuous chain: at the moment of the winning write, the engine recomputes this downpipe’s entire local chain into strict index order, atomically, inside the same conditional write. The signed root manifest still carries the global lineage. The effect is that failover never forges a forked or dangling chain, and a reader’s rollback check sees a clean, linear local chain wherever a run actually landed.

Where this fits

This page is the conceptual map. The detail and the doing live nearby.

Last updated .