Scale and limits: sliced runs, the slice budget, and the ceilings

The engine runs inside a Cloudflare Worker, and a Worker invocation has finite budgets: a cap on outbound subrequests and a ceiling on CPU and wall-clock time. A backup that cannot finish inside one invocation is not rejected and does not silently truncate. It is split into slices that each finalise within a per-invocation budget, and the run is carried across invocations on a durable-alarm chain. This page is for an operator or developer who needs the concrete figures: how big one slice gets, what one object can be, and how storage grows as runs accumulate.

It is the companion to the engine architecture, which owns the runtime model and the dispatch floor; this page owns the numbers. Each engine limit here names the constant that sets it.

Sliced runs: how a large backup crosses invocations

A backup run is the crawl of a source, the encryption of every record, and the write of an append-only archive to your destination. A small run completes inside the cron tick’s first inline slice, never persists a wrapped per-run key, and behaves exactly like a single-pass seal. A run that cannot complete in that one invocation is handed to a per-downpipe Durable Object, RunSealDO, which advances it one slice at a time.

Each alarm invocation of RunSealDO gets a fresh platform subrequest budget. That is the whole reason a large run is chained through alarms rather than looped inside one invocation that would hit the cap. On every alarm the object heartbeats the scheduler’s in-flight lease (so a long run is never reclaimed as crashed mid-flight), seals one budget’s worth of work, persists the advanced checkpoint and that slice’s output in one atomic write, then re-arms the next alarm. When the source is exhausted and there is headroom left, that same invocation finalises the run: it writes the root manifest, the signature and the recovery bundle, appends to the destination’s RUNLOG, reads the archive back to verify it, and posts the completion.

Two correctness properties hold throughout. The alarms are durable, so a slice that crashes resumes from its last checkpoint rather than restarting the whole run, and content addressing makes the re-seal idempotent (a segment already written is detected and skipped). The destination chosen when the run started is pinned for the life of that run, so a mid-run destination change can never split one archive across two stores.

The slice budget

SliceBudget (engine/src/seal/budget.ts) bounds one invocation’s share of the platform limits. The two limits that actually kill a seal are the per-invocation subrequest cap (every KV get, R2 get and put, S3 fetch and Durable Object fetch counts toward it) and the CPU and wall-clock. The budget is spent by the sources, the destination and the pipeline as they work; when it runs low the slice checkpoints between records and hands the run to the next invocation.

The defaults are deliberately conservative. The paid-plan subrequest cap is 1000, and a cron invocation also spends on the scheduler Durable Object, the notification passes and any other due downpipes, so a slice never assumes it owns the whole cap. The 700 default leaves room for that sharing and for the finalise reserve.

The reserve is derived from the worst-case sub-terms of the actual finalisation code paths, so that editing a retry count cannot silently break the arithmetic: the final flush, the root manifest, the signature and four recovery-bundle objects, each destination put retried up to three attempts (21); the RUNLOG append loop of up to six contended read-then-put rounds plus the re-read and the signature put (14); the RUNLOG lock acquire-and-release attempts (4); and the completion fetch (1). That sums to 40. The slice yields as soon as the remaining budget falls to the reserve or the wall-clock is consumed. It checks that boundary between records, and after each window inside a large streamed value that has a stable etag. The next slice resumes such a value from its last sealed window.

Tuning the budget

Two environment knobs override the defaults: SCALE_SLICE_SUBREQUESTS and SCALE_SLICE_WALL_MS. They are read per slice. An invalid value (not a positive number) falls back to the default rather than throwing, and an over-set value is clamped to the platform-survivable ceiling rather than honoured. This matters: past roughly 940 subrequests the yield signal could fire only after the 1000-cap had already killed the invocation, and a cap death persists no checkpoint progress, so an over-set knob would convert a large run into a permanently unfinishable one. The clamp keeps the yield able to fire before the platform cap. You should rarely need to touch either knob.

The segment and object ceilings

The archive is built from content-addressed segments. The hard ceiling is on one segment, not on one run.

A sealed segment streams to the destination as one object. No path splits an oversized segment into several objects, so a value is capped to MAX_STREAM_SEGMENT_BYTES before streaming, and the R2, S3 and Azure Blob destinations assert the same ceiling on the put as defence in depth. The Azure Blob destination asserts it from engine 0.3.6. (A maximal one-gibibyte plaintext segment seals to slightly more than a gibibyte once its per-chunk tags are added; the S3 path uploads that as roughly 33 parts of 32 MiB each.)

A value larger than the segment ceiling is not rejected outright, but it is not unbounded either. It is sealed as a chained multi-segment record: the value is windowed into one-gibibyte slices, each window addressed and sealed as its own segment, and the record line carries the ordered segment list with the whole-record plaintext digest and size. Both the standalone reader and the engine reassemble these chains on restore.

Byte sources (R2, Images, Stream, Artifacts) chain up to a named record ceiling, not indefinitely. A value with a stable etag can resume mid-record across slices, and its ceiling is MAX_SINGLE_RECORD_CONTENT_BYTES (roughly 3.66 GiB). A value with no stable etag must complete its chain inside one slice, and its ceiling is the lower MAX_SINGLE_SLICE_RECORD_BYTES (roughly 1.84 GiB). When SLICED_RUNS_DISABLED is set, every value has the lower ceiling. A value past its applicable ceiling is not sealed at all: it is skipped with a loud incompleteness marker (recordsIncomplete surfaces it; the rest of the backup completes), recoverable out of band with the downpipe CLI rather than wedging the run. Both ceilings are defined in engine/src/seal/budget.ts and applied in engine/src/sources/byte-fetch.ts and engine/src/sources/r2.ts.

D1 has no total-size cap. A D1 database is not buffered whole in memory. The dump is a header record with the table definitions, then one record for each keyset-ordered page of rows, then a schema record. The run can checkpoint after each page, except in the WITHOUT ROWID case below. Peak memory is one row page plus the current sealed chunk and the destination’s own write buffer (up to 32 MiB on S3), never the whole database, so a multi-gigabyte D1 is backed up rather than rejected.

There is no whole-body reject and no whole-export size gate: D1_EXPORT_SIZE_LIMIT (256 GiB) is a nominal constant that nothing enforces. The streamed total is bounded by the run’s segment and subrequest budget like every other source. The one live D1 ceiling is D1_PAGE_BYTE_LIMIT, a 96 MiB bound on one in-memory row page, refusing a single pathological row too large to window.

Up to engine 0.3.5, a WITHOUT ROWID table is read in one page, so that bound applies to the whole table. From engine 0.3.6, such a table pages by its PRIMARY KEY, and the bound applies to one page as for any other table. The run cannot checkpoint after a page whose last key is larger than 4 KiB.

Restore ceilings

A restore has its own bounds, separate from the seal-side segment ceiling above.

A record at or under the buffered ceiling is read whole and written in one pass. A larger R2 object streams to its destination and is read back afterwards to verify, which costs time, not memory. RESTORE_BUFFERED_MAX_BYTES can lower the buffered ceiling for a deployment that wants the streaming path more often; it can only lower it, never raise it past 32 MiB.

Neither ceiling bounds what a restore can recover. An R2 object too large to buffer streams instead. A media file too large to re-upload in-account is still recovered from the verified bytes for you to upload by hand. See media restore for the media path in full.

Storage growth: no cross-run deduplication

Within a single resumed run, content addressing deduplicates: a segment already written is skipped on the re-seal. There is no deduplication across runs. Each run derives a fresh per-run key, so its content addresses differ from every other run’s, and each run stores a full per-run snapshot.

The consequence for capacity planning is direct. Storage grows with how often you run, not with how much actually changed between runs. A daily downpipe over a mostly-static dataset still writes a full snapshot per day. Retention pruning is what bounds this over time: a superseded run’s tree is deleted and then the segments no retained run still references are garbage-collected, manifest-driven, so a delete only ever touches an object the planner has proven unreferenced. Size your destination and your retention window together, and see predicting cost for the model that turns run frequency and retention into a storage estimate.

The RUNLOG grows; the audit log is capped

Two logs grow as you operate, and they behave differently. Knowing which is bounded and which is not is the difference between a tidy account and a surprise.

The audit log is bounded. AUDIT_CAP is 10000 entries; at the cap the log rolls over the oldest entries first, and the console prompts an export as the count nears the cap so nothing is lost silently. The scheduled-restore drill-evidence log is bounded too, at DRILL_EVIDENCE_CAP of 500 rows, rolling oldest-first.

The RUNLOG is not bounded the same way. It is the signed, append-only record of every run on one destination, and it grows over time without a self-pruning cap of its own. It is small per entry, so this is slow, but on a long-lived account with many frequent downpipes it is a real, monotonic growth you should plan for rather than assume away. No setting trims it. If you need it smaller, the lever is run frequency.

The dispatch floor is a scale and scheduling consequence

The only thing that begins a new backup is the engine’s reconciliation cron, declared in engine/wrangler.toml as crons = ["*/15 * * * *"]. It ticks every fifteen minutes. The scheduler’s own alarm prunes expired records, re-arms, and runs the alert sweep when the cron goes silent. It does not dispatch a run. An already-started large run continues on the seal Durable Object’s alarm chain between ticks, but a new run only ever starts on a cron tick.

This is the source of the effective backup floor. Because the cron is the dispatch driver, ticking every fifteen minutes, a downpipe can actually begin a run only roughly every fifteen minutes. The console floors the schedule selector at Hourly rather than offer a finer cadence it could not keep. The fifteen-minute figure is the cron tick, not a tunable knob, so do not treat it as a recovery-point objective you can dial down. The freshness consequence is developed in full on objectives, RPO and RTO. The framing of the floor as a backup-scheduling property appears again on the backing-up overview.

Where this fits

For the runtime model behind slicing (the two Durable Objects, the lease, the two alarm chains), read the engine architecture. For what one run does end to end, see the anatomy of a backup run. For how often a backup can run and what that means for recovery, see objectives, RPO and RTO. For turning run frequency and retention into a storage and cost estimate, see predicting cost and retention and pruning.

Last updated 25 September 2026.