System update spec proposal

[Christopher Blizzard]

On Tue, 2007-06-26 at 18:50 -0400, C. Scott Ananian wrote:
> On 6/26/07, Christopher Blizzard <blizzard at redhat.com> wrote:
> > A note about the history of using rsync.  We used rsync as the basis for
> > a lot of the Stateless Linux work that we did a few years ago.  That
> > approach (although using LVM snapshots instead of CoW snapshots)
> > basically did exactly what you've proposed here.  And we used to kill
> > servers all the time with only a handful of clients.  Other people
> > report that it's easy to take out other servers using rsync.  It's
> > pretty fragile and it doesn't scale well to entire filesystem updates.
> > That's just based on our experience of building systems like what you're
> > suggesting here and how we got to where we are today.
> 
> I can try to get some benchmark numbers to validate this one way or
> the other.  My understanding is that rsync is a memory hog because it
> builds the complete list of filenames to sync before doing anything.
> 'Killing servers' would be their running out of memory.  Rsync 3.0
> claims to fix this problem, which may also be mitigated by the
> relatively small scale of our use:  my laptop's debian/unstable build
> has 1,345,731 files. Rsync documents using 100 bytes per file, so
> that's 100M of core required. Not hard to see that 10 clients or so
> would tax a machine with 1G main memory.  In contrast, XO build 465
> has 23,327 files: ~2M of memory.  100 kids simultaneously updating
> equals 2G of memory, which is within our specs for the school server.
> Almost two orders of magnitude fewer files for the XO vs. a 'standard'
> distribution ought to fix the scaling problem, even without moving to
> rsync 3.0.
>  --scott
> 

I think that in our case it wasn't just memory it was also seeking all
over the disk.  We could probably solve that easily by stuffing the
entire image into memory (and it will fit, easily) but your comment
serves to prove another point: that firing up a program that has to do a
lot of computation every time a client connects is something that's
deeply wrong.  And that's just for the central server.

Also, 2G of memory on the school server is nice - as long as you don't
expect to do anything else.  Or as long as you don't want to do what I
mention above and shove everything into memory to avoid the thrashing
problem.

--Chris