Upgrades and image manifests

[Dan Williams]

On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote:
> Hi, 
> 
> My name is Alexander Larsson, and I just started working on the field
> upgrade system of the olpc laptops. I have some ideas I'd like to
> explain and get feedback on.
> 
> The olpc uses a full-image system, as opposed to the per-package
> versioning scheme of deb or rpms. So, an upgrade consists of replacing
> your system files with a full copy of a later (or earlier) version of
> the system image. We want to support both upgrading from a central
> server or from another laptop, and we want to minimize the data we have
> to download for an upgrade.
> 
> To support this we create a manifest that describes each image, and we
> save a copy of the current image manifest installed on the laptop. The
> manifest format is very similar to the git "tree" object, and describes
> each file by listing name, metadata and sha1 hash value of the contents.
> 
> Here is a small example:
> ---------------------
> link 777 500:500 foo bar
> blob 664 500:500 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 changed.txt
> blob 664 500:500 65ef9261ad6cfaa8403f2fa66d8edfada1445e5a common
> blob 664 500:500 f660633c832712eba3c6edd39151987e46bc3f87 image1.txt
> blob 711 500:500 empty logfs.pdf
> dir 775 500:500 subdir1
> blob 664 500:500 empty subdir1/file1.txt
> blob 664 500:500 7448d8798a4380162d4b56f9b452e2f6f9e24e7a subdir1/file2.txt
> -------------------
> 
> Given a manifest for the current image and a manifest for the image to
> upgrade to it is very simple and cheap to calculate what changes you

So the manifest you update to is an interesting piece, and it's easier
because where you get it from is completely orthogonal to the update
process.  So, you could:

1) as others have suggested, XO periodically contacts the school server
2) XO sends it's current image, asks what to update to, if anything
3) school server makes policy decision about update
4) school server sends back new manifest
5) XO verifies manifests signature
6) XO downloads changesets

This is cool, because it doesn't matter _where_ you get the changesets
from, be it the school server (which should be seeding the changesets
anyway), or from another XO.  It optimizes for bandwidth, but not at the
expense of any security.  You already know what changesets are valid
based on the signature-verified server-provided manifest, and on the
self-verifying sha1 of the changesets that anyone can provide to you.  

All in all, simple, straightforward, and not very much code.

Dan

> need to make to transform the image (on the client), and what sha1 blobs
> you need to download. You can then download the file blobs from whatever
> source you have (they are self-verifying, since they are named by the
> sha1 hash of the content) and do the upgrade.
> 
> It is very simple to host the blobs. We just put them all in a directory
> named by sha1 and export that with a http server. One can also host
> multiple versions of an image in the same directory, while minimizing
> the space used for common data. (Possibly one might want to gzip the
> blobs too.)
> 
> I've written some code (attached) to generate and manipulate manifests
> like these. There are three tools:
> 
> * generate-manifest: This generates a manifest files given a path to a
> directory containing the image tree. You can also make it populate a
> directory of sha1-named blobs by giving it a blob directory with -b.
> 
> * diff-manifest: Gives a simple visual diff between two manifests. If
> you pass it -b it will instead give a list of all blobs required to
> update between the two manifests.
> 
> * upgrade-manifest: Updates an image from one manifest to another, given
> a path of a directory with the required blobs.
> 
> Using these tools I upgraded from
> olpc-redhat-stream-development-build-406-20070507_2141-devel_ext3-tree
> to olpc-redhat-stream-development-build-406-20070507_2157-ext3-tree in a
> directory on my development machine, so it seems to work so far.
> 
> For finding and downloading updates I was thinking of using avahi to
> publish the image id + version, and then downloading the data and
> manifests using http. That way that clients can detect machines around
> them with later versions and auto-download (and apply automatically or
> later). The individual laptops can publish the version of the image they
> are using, and a school server can publish several versions. From the
> point of view of the individual laptop searching for upgrades they will
> look the same. Of course, we'd have to sign the manifests with some key
> to make sure you can't just auto-upgrade any machine to a rouge image.
> 
> Now over to the questions I have:
> 
> I'd like to put this code in a repository somewhere. Where should I put
> it?
> 
> Does OLPC use selinux or xattrs? Because if so we have to extend the
> manifest format.
> 
> We need a library to do http downloads. I see we're shipping libcurl on
> the laptop. Is it ok to use this library? 
> (We also need a very simple http sever that maps sha1 -> file in system
> image for laptop-to-laptop upgrades, but I think we can do that without
> using a library.)
> 
> Is using avahi/mDNS on the mesh network ok?
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> Devel mailing list
> Devel at lists.laptop.org
> http://lists.laptop.org/listinfo/devel