Powered by

Pushing and pulling

Once you can snapshot a directory locally, the next step is moving snapshots to and from a store. snapdir splits this into four distinct verbs, and the distinction matters:

  • snapdir push — upload a snapshot (manifest + objects) to a store.

  • snapdir fetch — download a snapshot's objects from a store into the local cache. Nothing is materialized into a working directory.

  • snapdir checkout — materialize an already-available snapshot from the cache into a directory on disk.

  • snapdir pull — the convenience combination: fetch from a store and check it out to a path in one step.

In short: fetch gets objects into your cache, checkout writes them into a working tree, and pull does both. Reach for fetch when you want the data local but not unpacked (for example, to re-push it onward or to pre-warm a cache); reach for pull when you actually want the files on disk.

Every object is re-hashed against the manifest as it moves, so a successful transfer is also a proof of integrity.

Push a snapshot with snapdir push

push uploads a snapshot to the store given by --store and prints its ID. You can push a directory directly (snapdir snapshots it first) or push a snapshot you already staged by ID:

# Push a directory straight to a store.
id=$(snapdir push --store "file://$PWD/store" ./my-dir)

# Or push a previously staged snapshot by ID (omit the path).
snapdir push --store s3://my-bucket/snapshots --id "$id"

Pushes are content-addressed and incremental: objects already present in the store are skipped, so re-pushing after a small change only uploads the new objects. See snapdir push.

Fetch objects into the cache with snapdir fetch

fetch pulls a snapshot's manifest and objects from a store into the local cache and stops there — it does not write a working directory. Identify the snapshot with --id:

snapdir fetch --store s3://my-bucket/snapshots --id "$id"

After a fetch the snapshot is fully local: you can check it out later, verify it, or push it on to a different store without re-downloading. See snapdir fetch.

Check out a cached snapshot with snapdir checkout

checkout materializes a snapshot that is already available (typically because you fetched or staged it) into a destination directory:

snapdir checkout --id "$id" ./restored

By default files are copied out of the cache; pass --linked to hardlink them instead, which is near-instant and ideal for read-only working copies. Use --force to overwrite an existing destination. See snapdir checkout.

Do both at once with snapdir pull

pull is fetch + checkout: it downloads the snapshot from the store and checks it out to the given path in a single command. This is what you want for an ordinary restore:

snapdir pull --store "file://$PWD/store" --id "$id" ./restored

# Confirm the restore is byte-for-byte identical.
snapdir id ./restored   # prints the same $id

If snapdir id ./restored prints the same ID you started with, the restore is verified identical to the original. See snapdir pull.

Tuning transfers

push, fetch, pull, and checkout share the transfer-tuning flags:

  • -j, --jobs <N> — maximum concurrent object transfers. 0 (or auto) uses the number of CPUs, capped at 16. Also set via SNAPDIR_JOBS.

  • --limit-rate <RATE> — cap total bandwidth, wget-style, e.g. 10M, 512K, 1G (aggregate across all transfers). Also SNAPDIR_LIMIT_RATE.

  • --adaptive[=<FRACTION>] — opt in to adaptively tuning concurrency and bandwidth toward a fraction (default 0.8) of measured CPU/network capacity, backing off under contention. Default is full speed.

snapdir pull --store s3://my-bucket/snapshots --id "$id" -j 8 --limit-rate 20M ./restored

Where to go next