Upgrades and image manifests

[Dan Williams]

On Fri, 2007-06-29 at 12:26 -0400, C. Scott Ananian wrote:
> Please read Ivan's update proposal.  He addresses many of the
> peripheral issues that have arisen, re user notification, etc.
> 
> At this point, as far as I'm concerned, the only open question is how
> we get a filesystem image onto the laptop once we know which one to
> get.  Ivan suggests rsync; I refined that somewhat with a
> directory-at-a-time rsync using manifests; someone else (Mike?)
> proposed bittorrent, and previously I proposed a bespoke solution
> using binary diffs and multicast.
> 
> Alex's work is interesting, but we need to see benchmarks if we're not
> going to make a decision based on just fiat and personality.  I've
> posted up/download bandwidth for upgrades between 464-465 and 465-466.
>  Following up on Mike's suggestion, I've also patched bittorrent to
> allow using it for upgrades: the net bandwidth usage is something like
> 30MB, because it does not use block compression.  I can post more
> detailed comparisons if people are interested.  Please perform the
> equivalent benchmarks on the updatinator, before we continue this
> discussion.

With updatinator, bsdiff-ing the changes between files and gzipping new
files is the sweet spot.  I tested plain blobs, gzipped blobs, bzip2-ed
blobs, bsdiff + gzip, and bsdiff + bzip2.  Unfortunately, bzip2's memory
consumption on decompress isn't all that great according to other
people's research, and gzip gives us the best balance between
compression ratio and decompression mem/cpu usage.

Scott, you may also want to re-test the rsync benchmarks using rsync
compression to make the bandwidth numbers in the rsync benchmarks go
down.  You didn't mention anything specifically about using wire
compression, but the numbers look like you hadn't used it?

This patch adds bsdiff + gzip compression to updatinator:

http://people.redhat.com/dcbw/updatinator-bsdiff-gzip.patch

The difference sizes, using "du -csb" on the difference blob directory
are as follows.  This is the amount of data that would be transferred
over the network, not including HTTP headers.

464->465: 6,869,799 bytes
465->466: 18,870,574 bytes

The resulting image directories verify with both verify-manifest and
diff -rua.

As an improvement, we should provide a manifest-diff file for each
update path (along with the manifests for the actual image) that lists
the blobs with their own sha1 sum so that they are self-verifying.  This
would also simplify the patch quite a bit because there would be less
path-munging.  The diff-manifest tool that generates the blob diffs will
output this data in a somewhat suitable format already.

Cheers,
Dan

Procedure:

1) get the 465 and 466 "tree" files
2) unpack those trees
3) diff-manifest -g 465-466 465 466 465-manifest.txt 466-manifest.txt
4) cp -r 465 465-test
5) upgrade-manifest -b 465-466 -d 465-test 465-manifest.txt 466-manifest.txt
6) verify-manifest 466-manifest.txt 465-test