Powered by

Pushing and pulling

Once you can snapshot a directory locally, the next step is moving snapshots to and from a store. snapdir splits this into four distinct verbs, and the distinction matters:

  • snapdir push — upload a snapshot (manifest + objects) to a store.

  • snapdir fetch — download a snapshot's objects from a store into the local cache. Nothing is materialized into a working directory.

  • snapdir checkout — materialize an already-available snapshot from the cache into a directory on disk.

  • snapdir pull — the convenience combination: fetch from a store and check it out to a path in one step.

In short: fetch gets objects into your cache, checkout writes them into a working tree, and pull does both. Reach for fetch when you want the data local but not unpacked (for example, to re-push it onward or to pre-warm a cache); reach for pull when you actually want the files on disk.

Every object is re-hashed against the manifest as it moves, so a successful transfer is also a proof of integrity.

Push a snapshot with snapdir push

push uploads a snapshot to the store given by --store and prints its ID. You can push a directory directly (snapdir snapshots it first) or push a snapshot you already staged by ID:

# Push a directory straight to a store.
id=$(snapdir push --store "file://$PWD/store" ./my-dir)

# Or push a previously staged snapshot by ID (omit the path).
snapdir push --store s3://my-bucket/snapshots --id "$id"

Pushes are content-addressed and incremental: objects already present in the store are skipped, so re-pushing after a small change only uploads the new objects. See snapdir push.

Push into a shared object pool

If many manifest locations should share one deduplicated object pool, add --objects-store <URI> (or set SNAPDIR_OBJECTS_STORE) while keeping --store <URI> as the manifest-side location:

snapdir push \
  --store s3://inventory/manifests/host-a/2026-06-18 \
  --objects-store s3://inventory/object-pool \
  ./my-dir

In this mode, the manifest is stored under the --store location's .manifests/ tree, and file content objects are stored under the object pool's .objects/ tree. Re-pushing another manifest location against the same object pool only uploads changed objects.

Fetch objects into the cache with snapdir fetch

fetch pulls a snapshot's manifest and objects from a store into the local cache and stops there — it does not write a working directory. Identify the snapshot with --id:

snapdir fetch --store s3://my-bucket/snapshots --id "$id"

For a snapshot pushed with --objects-store, fetch with the same split:

snapdir fetch \
  --store s3://inventory/manifests/host-a/2026-06-18 \
  --objects-store s3://inventory/object-pool \
  --id "$id"

After a fetch the snapshot is fully local: you can check it out later, verify it, or push it on to a different store without re-downloading. See snapdir fetch.

Check out a cached snapshot with snapdir checkout

checkout materializes a snapshot that is already available (typically because you fetched or staged it) into a destination directory:

snapdir checkout --id "$id" ./restored

By default files are materialized as independent, editable files: snapdir uses a copy-on-write clone where the filesystem supports it, then falls back to a normal copy. Pass --linked to create a read-only symlink view into local content-addressed objects. Linked checkouts are zero-copy and useful for runtime inputs that should not be edited in place, but they require local objects; a remote store with --linked is refused. Use --force to overwrite an existing destination. See snapdir checkout.

Do both at once with snapdir pull

pull is fetch + checkout: it downloads the snapshot from the store and checks it out to the given path in a single command. This is what you want for an ordinary restore:

snapdir pull --store "file://$PWD/store" --id "$id" ./restored

# Confirm the restore is byte-for-byte identical.
snapdir id ./restored   # prints the same $id

If the snapshot's objects live in a shared pool, include --objects-store on the pull too. Pulling with the manifest store alone, or with the wrong object pool, cannot reconstruct the snapshot because the manifest location intentionally does not contain the objects:

snapdir pull \
  --store s3://inventory/manifests/host-a/2026-06-18 \
  --objects-store s3://inventory/object-pool \
  --id "$id" ./restored

If snapdir id ./restored prints the same ID you started with, the restore is verified identical to the original. See snapdir pull.

Keep a destination as an exact mirror

checkout and pull are additive unless you ask for mirror behavior. Existing files that are not in the snapshot are left alone by default. Add --delete to prune the destination to exactly the snapshot manifest:

snapdir pull \
  --store file:///srv/snapdir/features \
  --id "$id" \
  --linked --delete \
  ./runtime-features

The mirror pass is bounded to the destination root. It refuses dangerous destinations such as /, $HOME, the cache directory, or a store path even with --force, and it unlinks extraneous symlinks rather than following them outside the destination. Use --dryrun --delete to see the prune set first, and repeat --exclude <PATTERN> to protect local paths:

snapdir checkout --id "$id" --delete --dryrun ./runtime-features
snapdir checkout --id "$id" --delete --exclude '\.env$' ./runtime-features

When a long-running process already has a file open, normal POSIX inode behavior applies: replacing or pruning the path does not invalidate bytes the process already opened. The path on disk moves to the new snapshot; the old file descriptor keeps reading the old bytes until it closes.

Re-snapshotting linked views

A linked checkout stores each file as a symlink whose target is a local object addressed by BLAKE3. When snapdir id or snapdir manifest walks such a tree, it can reuse the checksum from that object path without reading the bytes again. That shortcut is checksum-only: a linked tree does not necessarily produce the same snapshot ID as the original source tree, because the walked symlink has its own size and permissions. Non-default checksum modes, keyed manifest contexts, and SNAPDIR_VERIFY_COPIES=1 fall back to normal content hashing or strict verification.

Local copy fast paths

When the source file and the local store or cache are on the same copy-on-write-capable filesystem, snapdir can avoid rewriting file bytes during stage, push, fetch, and checkout. On macOS it uses APFS clonefile(2); on Linux it uses the FICLONE reflink ioctl on filesystems that support it, such as Btrfs, XFS with reflinks, OpenZFS 2.2+, OCFS2, and bcachefs.

This is only a fast path. If the platform, filesystem, or cross-device layout cannot clone the file, snapdir falls back to a normal copy. Set SNAPDIR_CLONEFILE=0 to disable the clone/reflink attempt, or SNAPDIR_VERIFY_COPIES=1 to force the strict write-time re-hash even when a copy-on-write clone succeeds. Object bytes and snapshot IDs are unchanged with the fast path enabled, disabled, or verified strictly.

Tuning transfers

push, fetch, pull, and checkout share the transfer-tuning flags:

  • -j, --jobs <N> — maximum concurrent object transfers. 0 (or auto) uses the number of CPUs, capped at 16. Also set via SNAPDIR_JOBS.

  • --limit-rate <RATE> — cap total bandwidth, wget-style, e.g. 10M, 512K, 1G (aggregate across all transfers). Also SNAPDIR_LIMIT_RATE.

  • --adaptive[=<FRACTION>] — opt in to adaptively tuning concurrency and bandwidth toward a fraction (default 0.8) of measured CPU/network capacity, backing off under contention. Default is full speed.

snapdir pull --store s3://my-bucket/snapshots --id "$id" -j 8 --limit-rate 20M ./restored

Where to go next