XO in-field upgrades

[Mike C. Fletcher]

Alexander Larsson wrote:
> On Mon, 2007-06-25 at 15:38 -0400, Noah Kantrowitz wrote:
>   
...
> I still don't understand at all how vserver can be used to upgrade the
> main image. Yes, one can clone the main image into a chroot, and use
> vserver to boot into that. One can also enable some COW stuff so that we
> can change that into the new version. 
>
> But, how do we apply this new image (which is a copy of the root +
> applied updates) to be the new master image the next time we boot? i.e.
> when the kernel boots it will launch /bin/init,
> not /vserver/updated_image/bin/init.
>   
Any COW/overlay system used for the root file system requires a 
boot-time operation that sets up the root file system image, similar to 
an initrd based system.  This operation can be extremely simple with 
Ivan's two-version-only operation:

    if key_down( 'esc' ):
        load_readonly_root /rootfs/previous/bin/init
    else:
        load_readonly_root /rootfs/latest/bin/init +
    /rootfs/previous/bin/init

It's the same basic idea as a bootable CD where the initrd constructs a 
virtual root of a tmpfs and the CD image on disk, save that it allows 
for an interaction in which a keyboard operation changes the set of 
overlays used to construct the root fs.  Eventually there might be a UI 
for selecting overlays to strip them back one-by-one until you get a 
working system, but that's been tabled as an idea.

In the two-level approach, the upgrade system would have to first 
compress down the two levels of the current image before it began the 
update (since there's no provision for a third or higher level of overlay):

    * def update():
          o download_update into /roofts/unverified-version
          o if verify( /rootfs/unverified-version ):
                + assert no-activity-running-on-old-version
                + # merging moves the files from the one overlay
                  directory to the other
                + # applying deletions and the like. Can be atomic for
                  each file, hard-link
                + # the files rather than copying so that open files are
                  not disturbed
                + merge /rootfs/latest into /rootfs/previous
                + # make sure running activities still see the previous
                  version...
                + for each running core service and activity, remount
                  remount fs tree to remove /rootfs/latest
                      # (in aufs, for instance use the "del" option on
                        /rootfs/latest branch)
                + assert /rootfs/latest empty
                + move /rootfs/unverified-version to /rootfs/latest
          o else:
                + delete /rootfs/unverified-version

I may be missing an operation or two in there, but I *think* that's the 
intention.  That is, there's an ultimate loading level that sits outside 
the standard root to make it all work.  That loading level could be in 
the BIOS or on an initrd, but there does need to be a level to manage 
the ultimate overlay-based structure AFAIK.  And at normal boot, 
something has to set up the overlays, regardless of what overlay set is 
loaded.  That level of access is where the update manager has to sit to 
be able to do the updates AFAICS.  That is to accomplish a merge or 
remount of the core fs we need a way to ask the overlay manager to do 
some work for us at the update-manager level.
> Btw. I don't see why we would have to use vserver and COW for doing
> this, we just have to do the hardlink copy, and then apply the update on
> that, carefully always changing files so that hardlinks are broken (i.e.
> no in-place updates, just replace the files). However, that still gets
> us into the same situation, with a copy of the new image in a
> subdirectory that is not the main image.
>   
Yes, same basic effect, something outside the root needs to be able to 
switch to the root you choose, you could have a UI element that allows 
you to select the root, or you could have the service switch the name of 
the directories so that the new version is called "latest" and the last 
one "previous".  Basically the same problem, you need a way to ask 
something at a higher level of access to do the promotion.  The 
promotion could be something as simple as writing a file at the root 
that says "root=myversion" for the next boot, letting the boot-time 
process read that file in order to choose which root to boot.

BTW, reading the two descriptions, the hardlink-copy version is easier 
than the overlay AFAICS.  You don't have to remount the running 
services/activities (as you do if you require that there be a maximum of 
1 overlay).  Other than that the operations are essentially identical, 
you replace the merge with a recursive hard-linking copy (how does that 
work for soft links, incidentally, I gather you recreate them rather 
than hard-linking?)  Something at boot time then decides what to load, 
with some operation setting the default to the new version.  The 
advantages for the overlay system only start when you begin considering 
many small overlays, user-authored overlays and the like.

Anyway, just my fuzzy understanding,
Mike

-- 
________________________________________________
  Mike C. Fletcher
  Designer, VR Plumber, Coder
  http://www.vrplumber.com
  http://blog.vrplumber.com