Synchronizing xs-0.3 and xo-??? --- backups

[Tomeu Vizoso]

On Thu, Apr 24, 2008 at 6:18 AM, Martin Langhoff
<martin.langhoff at gmail.com> wrote:
>
>   - Initial HTTP check for protocol and XS availability stays. We drop
>  the 'nonce' value.
>   - Q: Any reason for this being REST vs XML-RPC for the registration service?
>
>   - Preparations are simpler
>    - No need for a master index. Always backup everything (let rsync
>  deal with what needs to be backed up), to a directory that will have
>  the last copy we rsync'd. Make extensive use of rsync smarts.
>    - We must dump _all_ the metadata to a CJSON file
>    - Q: Could we create a single metadata file per document (ahem,
>  Journal entry) instead?
>    - Q: Does it make sense to always maintain the CJSON-formatted
>  metadata anyway? (This would specially make sense if some of the DS
>  corruption reports being discussed are tracked down to Xapian.)

Extracting all the metadata from the index is a relatively expensive
process, that could affect very negatively the user experience while
it lasts. I really recommend to do it in smaller chunks (one hundred
entries?). One single metadata file per entry looks best to me, and is
already implemented here:

http://dev.laptop.org/git?p=users/krstic/ds-backup;a=summary

>    - Initially I thought that transferring the

Something missing here?

>  Full restore:
>
>  We can simplify things here
>   - If the XS is using rsync, the REST API returns a path, the XO
>  rsyncs from there
>   - If the XS is using git internally, there are more options it may
>  force a delay (to get a temp checkout that will be held there for
>  24hs) or perhaps the client can just execute git archive and pipe the
>  output to itself ;-)
>   - After retrieval of files from the XS, the XO must rebuild its
>  Xapian metadata store

Should this happen after each entry has been copied?

>   - Q: How do we trigger this from Sugar on a just-reflashed machine?

Which user experience do we want here? Eben?

>  Single-item-restore:
>
>  I'll punt on this in the short-term (as I have to prepare an XS
>  release too!), but various avenues are open. We can support web-based
>  zipfile download, specially since Tomeu's implemented the handling,
>  and we can support Journal-based browse-and-restore. But want to get
>  the major issues sorted first ;-)

Sure. Robson may be interested in lending a hand here. At any point,
thinking now about the different possibilities may not do any harm.

Thanks,

Tomeu