Upgrades and image manifests

[Alexander Larsson]

Hi, 

My name is Alexander Larsson, and I just started working on the field
upgrade system of the olpc laptops. I have some ideas I'd like to
explain and get feedback on.

The olpc uses a full-image system, as opposed to the per-package
versioning scheme of deb or rpms. So, an upgrade consists of replacing
your system files with a full copy of a later (or earlier) version of
the system image. We want to support both upgrading from a central
server or from another laptop, and we want to minimize the data we have
to download for an upgrade.

To support this we create a manifest that describes each image, and we
save a copy of the current image manifest installed on the laptop. The
manifest format is very similar to the git "tree" object, and describes
each file by listing name, metadata and sha1 hash value of the contents.

Here is a small example:
---------------------
link 777 500:500 foo bar
blob 664 500:500 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 changed.txt
blob 664 500:500 65ef9261ad6cfaa8403f2fa66d8edfada1445e5a common
blob 664 500:500 f660633c832712eba3c6edd39151987e46bc3f87 image1.txt
blob 711 500:500 empty logfs.pdf
dir 775 500:500 subdir1
blob 664 500:500 empty subdir1/file1.txt
blob 664 500:500 7448d8798a4380162d4b56f9b452e2f6f9e24e7a subdir1/file2.txt
-------------------

Given a manifest for the current image and a manifest for the image to
upgrade to it is very simple and cheap to calculate what changes you
need to make to transform the image (on the client), and what sha1 blobs
you need to download. You can then download the file blobs from whatever
source you have (they are self-verifying, since they are named by the
sha1 hash of the content) and do the upgrade.

It is very simple to host the blobs. We just put them all in a directory
named by sha1 and export that with a http server. One can also host
multiple versions of an image in the same directory, while minimizing
the space used for common data. (Possibly one might want to gzip the
blobs too.)

I've written some code (attached) to generate and manipulate manifests
like these. There are three tools:

* generate-manifest: This generates a manifest files given a path to a
directory containing the image tree. You can also make it populate a
directory of sha1-named blobs by giving it a blob directory with -b.

* diff-manifest: Gives a simple visual diff between two manifests. If
you pass it -b it will instead give a list of all blobs required to
update between the two manifests.

* upgrade-manifest: Updates an image from one manifest to another, given
a path of a directory with the required blobs.

Using these tools I upgraded from
olpc-redhat-stream-development-build-406-20070507_2141-devel_ext3-tree
to olpc-redhat-stream-development-build-406-20070507_2157-ext3-tree in a
directory on my development machine, so it seems to work so far.

For finding and downloading updates I was thinking of using avahi to
publish the image id + version, and then downloading the data and
manifests using http. That way that clients can detect machines around
them with later versions and auto-download (and apply automatically or
later). The individual laptops can publish the version of the image they
are using, and a school server can publish several versions. From the
point of view of the individual laptop searching for upgrades they will
look the same. Of course, we'd have to sign the manifests with some key
to make sure you can't just auto-upgrade any machine to a rouge image.

Now over to the questions I have:

I'd like to put this code in a repository somewhere. Where should I put
it?

Does OLPC use selinux or xattrs? Because if so we have to extend the
manifest format.

We need a library to do http downloads. I see we're shipping libcurl on
the laptop. Is it ok to use this library? 
(We also need a very simple http sever that maps sha1 -> file in system
image for laptop-to-laptop upgrades, but I think we can do that without
using a library.)

Is using avahi/mDNS on the mesh network ok?







-------------- next part --------------
A non-text attachment was scrubbed...
Name: manifest-0.1.tar.gz
Type: application/x-compressed-tar
Size: 9386 bytes
Desc: not available
URL: <http://lists.laptop.org/pipermail/devel/attachments/20070619/e46a9716/attachment.bin>