Synchronizing xs-0.3 and xo-??? --- backups
tomeu at tomeuvizoso.net
Mon Apr 28 10:14:54 EDT 2008
On Thu, Apr 24, 2008 at 6:18 AM, Martin Langhoff
<martin.langhoff at gmail.com> wrote:
> - Initial HTTP check for protocol and XS availability stays. We drop
> the 'nonce' value.
> - Q: Any reason for this being REST vs XML-RPC for the registration service?
> - Preparations are simpler
> - No need for a master index. Always backup everything (let rsync
> deal with what needs to be backed up), to a directory that will have
> the last copy we rsync'd. Make extensive use of rsync smarts.
> - We must dump _all_ the metadata to a CJSON file
> - Q: Could we create a single metadata file per document (ahem,
> Journal entry) instead?
> - Q: Does it make sense to always maintain the CJSON-formatted
> metadata anyway? (This would specially make sense if some of the DS
> corruption reports being discussed are tracked down to Xapian.)
Extracting all the metadata from the index is a relatively expensive
process, that could affect very negatively the user experience while
it lasts. I really recommend to do it in smaller chunks (one hundred
entries?). One single metadata file per entry looks best to me, and is
already implemented here:
> - Initially I thought that transferring the
Something missing here?
> Full restore:
> We can simplify things here
> - If the XS is using rsync, the REST API returns a path, the XO
> rsyncs from there
> - If the XS is using git internally, there are more options it may
> force a delay (to get a temp checkout that will be held there for
> 24hs) or perhaps the client can just execute git archive and pipe the
> output to itself ;-)
> - After retrieval of files from the XS, the XO must rebuild its
> Xapian metadata store
Should this happen after each entry has been copied?
> - Q: How do we trigger this from Sugar on a just-reflashed machine?
Which user experience do we want here? Eben?
> I'll punt on this in the short-term (as I have to prepare an XS
> release too!), but various avenues are open. We can support web-based
> zipfile download, specially since Tomeu's implemented the handling,
> and we can support Journal-based browse-and-restore. But want to get
> the major issues sorted first ;-)
Sure. Robson may be interested in lending a hand here. At any point,
thinking now about the different possibilities may not do any harm.
More information about the Devel