Synchronizing xs-0.3 and xo-??? --- backups

[Martin Langhoff]

Tomeu, Ivan,

with my notes from the last software-status meeting and a few other
bits and pieces, my plan is to follow roughly the spec but with a few
changes.

Making a backup:

 - Initial HTTP check for protocol and XS availability stays. We drop
the 'nonce' value.
 - Q: Any reason for this being REST vs XML-RPC for the registration service?

 - Preparations are simpler
   - No need for a master index. Always backup everything (let rsync
deal with what needs to be backed up), to a directory that will have
the last copy we rsync'd. Make extensive use of rsync smarts.
   - We must dump _all_ the metadata to a CJSON file
   - Q: Could we create a single metadata file per document (ahem,
Journal entry) instead?
   - Q: Does it make sense to always maintain the CJSON-formatted
metadata anyway? (This would specially make sense if some of the DS
corruption reports being discussed are tracked down to Xapian.)
   - Initially I thought that transferring the

 - Control moves from HTTP-based (with the 'nonce' identifier) to an
rsync wrapper on the XS side that...
    - re-checks 'availability/load' criteria
    - uses a lock to prevent overapping transfers from the same client
    - sets a flag to indicate that the transfer was successful for
XS-side post-processing

 - XS postprocessing - once a backup is successful, we'll make a
hardlinked copy to a timestamped directory. rsync is good about
breaking hardlinks, so this allows us to do a cheap storage in a
pdumpfs fashion. In any case, this is an XS-side thing - might do it
with git too ;-)

Full restore:

We can simplify things here
 - If the XS is using rsync, the REST API returns a path, the XO
rsyncs from there
 - If the XS is using git internally, there are more options it may
force a delay (to get a temp checkout that will be held there for
24hs) or perhaps the client can just execute git archive and pipe the
output to itself ;-)
 - After retrieval of files from the XS, the XO must rebuild its
Xapian metadata store
 - Q: How do we trigger this from Sugar on a just-reflashed machine?
 - Q: We'll have to define an interim means of aliasing the acct so
that the XS allows restore to a different XO in case of a replaced XO.

Single-item-restore:

I'll punt on this in the short-term (as I have to prepare an XS
release too!), but various avenues are open. We can support web-based
zipfile download, specially since Tomeu's implemented the handling,
and we can support Journal-based browse-and-restore. But want to get
the major issues sorted first ;-)

cheers,



m
-- 
 martin.langhoff at gmail.com
 martin at laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff