System update spec proposal

[tridge at samba.org]

Ivan,

 > I'm curious: would it be easier if we had rsync write out the in- 
 > memory file list to disk after a successful request (named, as you  
 > proposed, after a hash of the options), and to simply monkeypatch the  
 > file list generation routine to check whether there's a matching on- 
 > disk list before agreeing to generate one from scratch?

yes, although I was thinking of having a command line option to rsync
to do the generation from cron (as you probably don't want the file to
be writable from within rsync after it does its setuid and chroot).

 > Seems like it might be a pretty simple hack for someone who's
 > familiar with rsync, but then I have little knowledge of rsync
 > internals. I also suspect it might not be a total hack -- anyone
 > serving large trees that change relatively infrequently could
 > benefit from the functionality.

right - basically hack flist.c which does the file list generation,
and in particular put a wrapper around send_file_list(). 

The trick is coping with all the places that the rsync server wants to
write to this list during a run. To cope with that we could:

 1) use MAP_PRIVATE for the mmap, so if the server does write it
 doesn't segv, it just gets copy on write on those pages. We need to
 make sure that doesn't happen too often though, or we end up chewing
 memory again.

 2) make sure global operations on the file list (like the qsort() in
 fsort()) either get disabled, or don't end up actually changing
 memory. This needs to be done very carefully though, as if the two
 ends of the link disagree about the sorting of the list then you get
 file corruption (rsync will update the wrong file!)

Internally rsync is a pretty delicate piece of code (sorry about
that), and it would be easy to introduce a subtle bug that corrupted
files. The basic idea of mmap hack is certainly good, I am just wary
of trying to say how long it would take to get right.

Wayne may have some useful comments, as he's been the one maintaining
rsync for the past few years.

Cheers, Tridge

PS: I'm away for a few days this week, so expect slow responses from
me until next week