Fwd: XO in-field upgrades

[C. Scott Ananian]

[grumble, sent this from the wrong email address.]

---------- Forwarded message ----------
Here are some quick numbers justifying the use of binary diffs as an
upgrade delivery format.  I just grabbed the latest debian security
advisory for Etch, which happened to be:
http://www.us.debian.org/security/2007/dsa-1320 on clamav.
This upgrade appeared not atypical, but feel free to point out if it
is widely uncharacteristic for a lightweight package (ie, not
mozilla).

I downloaded and unpacked the packages corresponding to the 'before'
and 'after' versions of all the packages affected.  For reference,
those versions are 0.90.1-3etch2 and 0.90.1-3etch3.

I then deleted the files in /usr/share/doc and /usr/share/man (neither
of which 'should' have changed, but apparently they did) and
/usr/lib/debug.

I wrote a script to generate binary diffs using both xdiff and bsdiff
for the files which changed.  (I also munged slashes to dashes.)
These files were:

usr-bin-clamconf   usr-bin-freshclam    usr-lib-libclamav.so.2.0.1
usr-bin-clamdscan  usr-bin-sigtool      usr-sbin-clamav-milter
usr-bin-clamscan   usr-lib-libclamav.a  usr-sbin-clamd

For all files other than libclamav.a and libclamav.so.2.0.1, the
binary diff was less than 164/225 bytes (xdiff/bsdiff).  libclamav.a
had a 3529/6571 byte diff, and libclamav.so.2.0.1 had a 13398/74461
byte diff.  (Note larger diff because all the relocation table entries
moved.  Maybe a smarter diff is possible.)

In comparison, the full libclamav.a blob is 567,140 bytes and
libclamav.so.2.0.1 is 432,668 bytes.

Details:
blob:
 17168 usr/bin/clamconf
 33120 usr/bin/clamdscan
 49600 usr/bin/clamscan
 65584 usr/bin/freshclam
 77868 usr/bin/sigtool
567140 usr/lib/libclamav.a
432668 usr/lib/libclamav.so.2.0.1
 76260 usr/sbin/clamav-milter
 49948 usr/sbin/clamd

bdiff:
  158 usr-bin-clamconf
  158 usr-bin-clamdscan
  161 usr-bin-clamscan
  159 usr-bin-freshclam
  162 usr-bin-sigtool
 3529 usr-lib-libclamav.a
13398 usr-lib-libclamav.so.2.0.1
  164 usr-sbin-clamav-milter
  162 usr-sbin-clamd

Total bsdiff size: 18,051
Total raw size: 1,369,356 bytes.  (>75x larger)

Further, although my goal of single packet upgrades seems
overoptimistic, all but two of the file diffs can fit in a single
packet, helping assure atomicity.  The total upgrade information fits
in 13 packets.

To my mind, this makes it clear that the on-wire format should be a
binary diff.  We could uncompress these diffs on receipt and maintain
a blob store as in the current proposal, but it seems much more
reasonable to sign, authenticate, and store the *diffs* rather than
the *blobs*, so that we don't need to recompute diffs when we want to
pass on our upgrade to a neighboring XO.
 --scott
--
                         ( http://cscott.net/ )


-- 
                         ( http://cscott.net/ )