Upgrades and image manifests

[Dan Williams]

On Tue, 2007-06-19 at 14:39 +0200, Alexander Larsson wrote:
> Hi, 
> 
> My name is Alexander Larsson, and I just started working on the field
> upgrade system of the olpc laptops. I have some ideas I'd like to
> explain and get feedback on.
> 
> The olpc uses a full-image system, as opposed to the per-package
> versioning scheme of deb or rpms. So, an upgrade consists of replacing
> your system files with a full copy of a later (or earlier) version of
> the system image. We want to support both upgrading from a central
> server or from another laptop, and we want to minimize the data we have
> to download for an upgrade.
> 
> To support this we create a manifest that describes each image, and we
> save a copy of the current image manifest installed on the laptop. The
> manifest format is very similar to the git "tree" object, and describes
> each file by listing name, metadata and sha1 hash value of the contents.
> 
> Here is a small example:
> ---------------------
> link 777 500:500 foo bar
> blob 664 500:500 4e1243bd22c66e76c2ba9eddc1f91394e57f9f83 changed.txt
> blob 664 500:500 65ef9261ad6cfaa8403f2fa66d8edfada1445e5a common
> blob 664 500:500 f660633c832712eba3c6edd39151987e46bc3f87 image1.txt
> blob 711 500:500 empty logfs.pdf
> dir 775 500:500 subdir1
> blob 664 500:500 empty subdir1/file1.txt
> blob 664 500:500 7448d8798a4380162d4b56f9b452e2f6f9e24e7a subdir1/file2.txt
> -------------------
> 
> Given a manifest for the current image and a manifest for the image to
> upgrade to it is very simple and cheap to calculate what changes you
> need to make to transform the image (on the client), and what sha1 blobs
> you need to download. You can then download the file blobs from whatever
> source you have (they are self-verifying, since they are named by the
> sha1 hash of the content) and do the upgrade.

Two questions here:

1) what does the scheme do in the case where the file it's about to
replace on the local machine isn't the same as what the manifest on the
local machine says?  ie, local changes have changed the sha1 hash of the
local file.  That's essentially a policy decision, do we replace when
there are local changes or don't we.  A %config-type thing could work
here to mark files that you may/may not always want to replace.

2) After downloading a blob, I assume the tool sha1s the downloaded file
to ensure that it's content matches the name, right?


> It is very simple to host the blobs. We just put them all in a directory
> named by sha1 and export that with a http server. One can also host
> multiple versions of an image in the same directory, while minimizing
> the space used for common data. (Possibly one might want to gzip the
> blobs too.)
> 
> I've written some code (attached) to generate and manipulate manifests
> like these. There are three tools:
> 
> * generate-manifest: This generates a manifest files given a path to a
> directory containing the image tree. You can also make it populate a
> directory of sha1-named blobs by giving it a blob directory with -b.
> 
> * diff-manifest: Gives a simple visual diff between two manifests. If
> you pass it -b it will instead give a list of all blobs required to
> update between the two manifests.
> 
> * upgrade-manifest: Updates an image from one manifest to another, given
> a path of a directory with the required blobs.
> 
> Using these tools I upgraded from
> olpc-redhat-stream-development-build-406-20070507_2141-devel_ext3-tree
> to olpc-redhat-stream-development-build-406-20070507_2157-ext3-tree in a
> directory on my development machine, so it seems to work so far.
> 
> For finding and downloading updates I was thinking of using avahi to
> publish the image id + version, and then downloading the data and
> manifests using http. That way that clients can detect machines around
> them with later versions and auto-download (and apply automatically or
> later). The individual laptops can publish the version of the image they
> are using, and a school server can publish several versions. From the
> point of view of the individual laptop searching for upgrades they will
> look the same. Of course, we'd have to sign the manifests with some key
> to make sure you can't just auto-upgrade any machine to a rouge image.
> 
> Now over to the questions I have:
> 
> I'd like to put this code in a repository somewhere. Where should I put
> it?
> 
> Does OLPC use selinux or xattrs? Because if so we have to extend the
> manifest format.
> 
> We need a library to do http downloads. I see we're shipping libcurl on
> the laptop. Is it ok to use this library? 

Yeah, though the ironic thing just occurred to me that libcurl is (along
with Mozilla) one of the more frequently updated pieces of a linux
distro :)

> (We also need a very simple http sever that maps sha1 -> file in system
> image for laptop-to-laptop upgrades, but I think we can do that without
> using a library.)

"simple" means lighttpd in my mind, but apache has a longer track
record.

> Is using avahi/mDNS on the mesh network ok?

Certainly.  There is no other way to do link-local presence discovery.
avahi can be controlled either via D-Bus or through a socket-based
interface, you are looking for the EntryGroup object.  The avahi Python
examples (see the Fedora avahi-tools package) are a good map the of the
D-Bus API.

Dan