Powered by

Syncing

snapdir sync copies a snapshot — its manifest and every object it references — directly between two stores, streaming through memory with no local staging. This is how you replicate a snapshot from one bucket or region to another (or between clouds) without ever materializing it on your own disk.

It is the right tool when the data does not need to land locally: a normal fetch + push would pull every object into your cache and push it back out, whereas sync pipes the snapshot straight from the source store to the destination store.

Replicate a snapshot between stores

sync requires a snapshot --id and both endpoints, --from and --to:

snapdir sync \
  --id "$id" \
  --from s3://primary-bucket/snapshots \
  --to   gs://dr-bucket/snapshots

When omitted, --from defaults to $SNAPDIR_STORE, so you can set the source store once via the SNAPDIR_STORE environment variable and only pass --to on each sync. --to is always explicit — a sync needs two distinct stores, so the destination is never inferred.

The source and destination must differ. The transfer is content-addressed, so objects already present at the destination are skipped — re-running a sync after a small change only copies the new objects, which makes sync cheap to run repeatedly as a replication step.

Both endpoints must be in-process stores (file://, s3://, gs://, b2://). External snapdir-*-store URLs have no in-process streaming surface and are rejected by sync; to replicate through a custom backend, fall back to a fetch then push. See Stores for the backend matrix and authentication.

Tuning and dry runs

sync honors the shared transfer-tuning flags, applied to the single store-to-store pipe:

  • -j, --jobs <N> — concurrent object transfers (0/auto = CPUs, capped 16).

  • --limit-rate <RATE> — cap aggregate bandwidth, e.g. 50M, 512K, 1G.

  • --adaptive[=<FRACTION>] — adaptively tune concurrency/bandwidth toward a fraction (default 0.8) of measured capacity, backing off under contention.

  • --dryrun — report what would be copied without writing anything to the destination. Useful for confirming a replication plan before it runs.

# Preview a cross-cloud replication, then run it rate-limited.
snapdir sync --id "$id" --from s3://primary/snap --to b2://dr/snap --dryrun
snapdir sync --id "$id" --from s3://primary/snap --to b2://dr/snap --limit-rate 50M

On a real sync the snapshot ID is printed to stdout and a human-readable summary goes to stderr, consistent with push and stage.

Where to go next