Skip to content
downpipes docs

Snapshot consistency: live crawls versus a point-in-time view, and what a downpipes backup actually represents

There is no whole-account point-in-time snapshot. A downpipes backup of KV or R2 is a live crawl of a store that may be changing while the crawl runs, so a single backup can mix keys that were written at slightly different instants. This page explains that reality in plain terms, says exactly which one source does give a coherent cross-table snapshot and under what condition, and defines what “skipped” means so you do not read it as a failure. It is written for a developer who needs to know what guarantee a backup carries before relying on it.

The honest headline is that a backup is consistent per record, not consistent across the whole account at a single instant. For most KV and R2 workloads that is exactly what you want and exactly what a periodic backup can promise. Where you need cross-record consistency, D1 gives it, and the practical reading at the end of this page tells you how to lean on that.

KV is a live crawl

A KV backup lists the namespace and reads each in-scope key’s value. KV has no point-in-time snapshot to read against, so this is a crawl over a live namespace. A key that is deleted between the moment it appeared in a list page and the moment the crawl reaches it reads back as gone, and the crawl simply skips it. That is the designed behaviour, not an error.

The crawl is ordered. KV list results are lexicographic, which is what lets a large namespace be backed up across more than one engine invocation: each fully-yielded page records a resume watermark (its last key), and a resumed crawl continues after that key without re-reading any value. So while a KV backup is not a point-in-time view, it is a single ordered pass in which each key is read once.

R2 is a live ordered crawl too

R2 behaves the same way, with one extra safeguard on its large objects. The bucket is listed in lexicographic order and each in-scope object is read. A streamed object over 8 MiB is read twice (once to address it, once to seal it), so each pass of that streamed read is pinned to the etag observed at list time, because without the pin an object replaced between those two reads could seal bytes that do not match the recorded address. With the pin, a replaced object reads as gone, exactly the same shape as a vanished key, and the writer skips the record loudly rather than archiving something that can never verify. A smaller object is read with a single plain get, so it is one read that cannot tear and needs no pin.

Object size decides how an object is read. An object over 8 MiB is streamed: it is read in passes and fed into the seal in windows, never held whole in memory. A smaller object is read whole, which is simpler and deduplicates the same way. Neither path buffers a large object.

Skipped is expected, not data loss

When the engine skips a key or object that vanished or changed mid-crawl, the run does not fail and the record is excluded from the archive entirely. This is the correct outcome for a live store: the alternative would be archiving a value that no longer exists, or one whose bytes do not match its recorded address. A skip is an honest in-flow outcome.

D1 is the one consistent point-in-time view

D1 is the exception, and it is a real one.

Record structure

A D1 database is backed up as a resumable sequence of records, not one whole-database record: one header record carrying every table’s CREATE statement and column names with no rows, then many per-table keyset row-page records (bounded by a per-page byte target of 8 MiB, a hard cap of 96 MiB, and up to 2,000 rows per page), then one schema record carrying the CREATE INDEX, TRIGGER and VIEW statements, applied last. A legacy encoder that wrote one whole-database record still exists so that archives written before this sequence keep restoring, but no new crawl produces one.

Session-bookmark mechanism

The whole sequence reads through one D1 Session anchored with withSession("first-primary"). The first query goes to the primary, which holds the newest committed state, and establishes a bookmark; every later read on that session, across every record in the sequence and across every resumed slice, is constrained to a replica at or after that bookmark. The header, every row page and the schema record therefore see one sequentially-consistent point-in-time snapshot. A database under concurrent write load is exported torn-free, rather than mixing pre-write and post-write rows across tables.

Each record is produced incrementally, not buffered as one dump. The plan (every table’s DDL and columns, plus the index/trigger/view text) is read once up front, on the first query, which is also what establishes the session bookmark described above. The header record is built from that plan and emitted first; each table’s rows are then read in keyset-ordered pages (paging by rowid where the table has one) and emitted one page at a time; the schema record is built from the same plan but emitted last, after every row page, so a restore can create the tables, load the rows, then add the indexes and triggers. Records are fed into the seal as a re-openable stream, the same way a large R2 object is sealed in windows, so peak memory is one row page plus the current sealed chunk and a multi-gigabyte database is backed up rather than rejected. An older note describing a buffered one-gibibyte cap is stale; that whole-body reject is gone.

The D1 snapshot is conditional

The D1 snapshot’s cross-table consistency is not unconditional in every environment: it holds on the production D1 binding, which supports withSession. On an older or local binding that does not expose withSession, the export degrades gracefully to a best-effort read against the bare database: the reads are still ordered keyset pages, but the single cross-table snapshot guarantee is then best-effort rather than sequentially consistent. So the guarantee is “sequentially consistent on the production binding, best-effort without withSession”, never “always a perfect snapshot everywhere”.

What is buffered, and what is not

It is worth drawing the line plainly, because “streamed” is the rule and the exception is narrow.

Capture Held whole in memory?
A large R2 object (over 8 MiB) No, streamed in windows
A small R2 object Yes, read whole (it is small)
The D1 header, row-page and schema records No, produced and sealed incrementally, one record at a time
A Worker script body Yes, buffered, capped at 128 MiB

The Worker script body is the deliberate buffered exception: a script past the cap is marked unavailable rather than risking the isolate.

Where you see skipped counts

The console treats a skip as ordinary. When you run a backup now from the downpipes screen, the trigger surface states that a “skipped” outcome is expected in flow and not an error, and a coalesced trigger (a run already in progress) is reported as skipped in the same calm way. So the word appearing in a run summary or a toast is the system being honest about a live crawl, not a warning that something broke.

How this connects to recovery

Because there is no within-run point-in-time view for KV and R2, recovery is at run granularity, not at an arbitrary instant inside a run. When you choose a moment to restore to, downpipes resolves it to the latest successful run that completed at or before that moment, read from a bounded per-downpipe history ring. If the moment you pick is older than the oldest run the ring still holds, the engine says so plainly rather than inventing a recovery point. There is no arbitrary-time picker, because there is no continuous timeline to pick from: the recovery points are your retained runs. See the recovery overview and restore flow for how a run is chosen and replayed.

A short practical reading

Two habits follow from the difference between a live crawl and a sequentially-consistent snapshot. Where you need several records to be consistent with each other, put that data in D1 and lean on its sequentially-consistent snapshot, which is the strongest cross-record guarantee downpipes offers. For KV and R2, treat each backup as a consistent-per-key ordered crawl rather than a global instant, and lean on cadence and on failover copies across destinations so that a key caught mid-change in one run is captured cleanly in the next.

Where this fits

Last updated .