rsync benchmarking

[C. Scott Ananian]

I wrote a quick ~60 line script to do non-recursive rsync a directory
at a time.  Actually, it's a little smarter than that: it generates
manifests for each directory, and syncs the tree by first syncing the
root (and the root's manifest).  It then checks the directory hashes
in the roots manifest against the current contents of the directory's
manifest, and recurses only if they don't meet (ie, if the directory
has been updated).

Anyway, here are some quick stats, for upgrades from build 464 to 465 to 466:

464-to-465:
Full contents of tree: 429,302,610 bytes
rsync --whole-file: sent 32,034 bytes; received 13,153,813 bytes.
Standard rsync: sent 96,720 bytes; received 10,047,524 bytes. 9.0s
user, 64.9s real
Rsync-with-manifest: sent 79,192 bytes; received 9,139,515 bytes. 4.3
user, 11.5s real.

465-to-466:
Full contents of tree: 458,828,206 bytes
rsync --whole-file: sent 21,990 bytes; received 25,826,516 bytes.
Standard rsync: sent 141,948 bytes; received 23,981,423 bytes. 14.0s
user, 68.5s real
Rsync-with-manifest: sent 125,158 bytes; received 23,171,430 bytes.
10.4 user, 28.5s real.

Using manifests saves a bit of bandwidth but a lot of memory and disk
I/O.  I'll put the python scripts I used to create, verify, and sync
with manifests online soon.
 --scott

-- 
                         ( http://cscott.net/ )