update mechanism for new releases

[Peter Robinson]

> We have a bug open here which is a difficult question, in my opinion:
> http://dev.laptop.org/ticket/9349
>
> For the upcoming XO-1.5 software build (which will hopefully make it
> onto XO-1 in the near future), do we want to ship olpc-update or use
> standard Fedora technologies?
>
>
> My unconclusive thoughts and questions:
>
>
> olpc-update has some features which aren't available elsewhere, such as
> the ability to switch between 2 OS builds installed on the same disk.
> However, I've never seen or heard of this being used in the field.
>
> olpc-update is a "deployment quality" system in that it has covered
> everything - not only from being an updater, it has olpc-update-query
> which implements logic to figure out when to ask for updates and where
> to ask for them. It's accompanied by the theft deterrence protocol and 3
> different server-side implementations of that. It's been implemented in
> OLPC deployments.
>
> olpc-update creates a huge mass of hardlinks on disk, one for every file
> on the main OS. It then rsyncs in the updated files on a copy-on-write
> basis. Therefore you end up with a hell of a lot of hardlinks, and 2
> complete copies of every file that has changed. (this means it is not
> great for situations when a lot of files have changed, e.g. changing
> between major Fedora releases)
>
> olpc-update presents challenges for deployments that customise their
> images. Namely, to produce a build that it is possible to olpc-update to
> involves quite a bit of hassle (inserting security keys into the
> firmware of each XO, setting up an OATS server, etc). We did manage to
> figure this out in Paraguay (thanks to the internet connection, since it
> wouldn't be safe to put the OATS server in the schools) and I improved
> some tools accordingly.
>
> We've switched to dracut for our initramfs, meaning that the
> initramfs-level stuff for the olpc-update system needs to be
> reimplemented. Basically we have to wrangle with the strange /versions
> filesystem layout and "frob" the /versions/current symlink if the user
> is switching between versions. I have reimplemented this logic in a
> dracut module but it is completely untested.
>
>
>
> The option for using "Standard Fedora technologies" as far as I am aware
> is to basically use "yum update" in a fashion somehow suited for
> deployments.
>
> Before F11, this would have meant a lot of downloading as each RPM
> update was downloaded in full even if only a little bit changed. As of
> F11, Fedora is now joining the "only download the bits that change"
> movement with yum-presto and deltarpms: http://lwn.net/Articles/329484/
>
> The deltarpm description on that lwn article leaves a little to be
> desired -- it wouldn't work with big packages (is the quoted 200mb the
> compressed or uncompressed size?), the reconstruction from deltas is a
> CPU intensive task and we have a slow CPU, and it's all done in memory
> and we don't have a huge amount of memory especially on XO-1.
> olpc-update has an overhead of about 5 minutes of checksumming files.

Well it works fine for OpenOffice and that's probably about the
biggest package I have on my F-11 system. The file size might be a
restriction to keep the memory usage in check. The thing with deltarpm
is ultimately its a decision between bandwidth usage and memory usage.
If all the updates are coming from a local school server and there's
only a few updates a month its probably better going with the complete
rpms. If bandwidth is an issue then drpms might well be the way to go.
Fortunately its easy to enforce from the server side so you could
include the deltrarpm support on the client side and if its not wanted
just don't make the drpms available from the server.

> Even though olpc-update isn't great for doing big distro updates
> (because of storing 2 copies of changed files, in this case almost all
> of them), it worked in those situations. I've never attempted an
> RPM-based update from e.g. Fedora 10 to Fedora 11. How well does that
> work out for regular Fedora users?

I've never had a problem with it. I've used it to remotely update my
australian server from F-7 -> F-11 (not at once), and never had an
issue with it. I've done it without issue on my local London firewall
as well (a Fit-PC which runs the same style Geode a the XO). The main
issue is that Fedora only tests upgrades between supported versions.
So they "support" yum upgrades to F-11 from F-9 and F-10.

> "yum update" always seems to use a surprising amount of bandwidth,
> redownloading entire package databases just to see if anything new is
> available. olpc-update was much more efficient in this respect, sending
> only a 128-byte hash of the filesystem contents file to the OATS server.
> For rpms, is there a more efficient alternative for updates-checking in
> situations where there is only going to be e.g. 1 update per month?

This should have improved greatly from F-9 to F-11. As for more
efficient, not that I'm aware of but if there's only one update a
month why does it matter?

> "yum update" has historically not worked very well on the XO. It hits
> OOM, it fills up yum's cache and then aborts, it gets confused between
> i586 and i686, etc.

That should be a thing of the past with F-11. Also its a simple change
to a rpm config file to change the geode from being a i586 to i686 as
far as rpm is concerned. With F-11 being i586 this also shouldn't
really be an issue either and the problem will be gone entirely with
F-12. rpm and yum memory usage has improved alot in recent releases so
hopefully the overall situation should be improved.

http://laiskiainen.org/blog/?p=19

> We would have to tweak yum's behaviour quite heavily.. I don't think we
> want an rpm cache, or for it to keep the .rpm files at all.

It only keeps the rpms until its installed them. Most of that should
be easily tunable through a olpc-relase package similar to or that
depends on the generic-release or fedora-release package.

> It introduces new questions of security, signing, etc. Deployments will
> want to sign their rpm updates that they push out, so we now need a
> mechanism for getting that specific RPM signing public key onto all the
> laptops in a deployment so that they can trust the updates server.
> We had this nailed down for olpc-update: deployments can insert "local"
> public keys into the manufacturing through keyjector firmwares for
> existing laptops, and Quanta can now manufacture laptops with these keys
> already in place for future orders. olpc-update and the bitfrost code
> used these keys to verify updates.

The way to get the signing key onto them would be through or in the
equivalent of the fedora-release file.

> For the XO-1 possibility it raises the question of how existing laptops
> could be migrated to this new system, without losing their user data.

I think It would be possible but obviously something that needs
testing, and obviously would be dependant partially on the deployment
as well.

> Using "yum update" would be a large benefit of sharing technologies with
> other communities and not having to maintain our own systems and tools.
> However olpc-update would be easier in the short term because all the
> components have been implemented and deployed, and there aren't really
> any open questions.

Agreed.

> Any other thoughts/opinions one way or the other?

None either way, obviously using yum has the advantage of being
upstream and having mainstream testing and development. The advantage
of olpc-update is that it has specific features, some which are used,
some aren't and was designed with olpc deployments in mind. Maybe some
of that functionality can be implemented through the yum plugin
system.

Peter