Synchronizing xs-0.3 and xo-??? --- backups

[Martin Langhoff]

On Tue, Apr 29, 2008 at 2:14 AM, Tomeu Vizoso <tomeu at tomeuvizoso.net> wrote:
>  >    - Q: Does it make sense to always maintain the CJSON-formatted
>  >  metadata anyway? (This would specially make sense if some of the DS
>  >  corruption reports being discussed are tracked down to Xapian.)
>
>  Extracting all the metadata from the index is a relatively expensive
>  process, that could affect very negatively the user experience while
>  it lasts.

Is it? Ugh then. (I thought it'd be a cheap op!)

> I really recommend to do it in smaller chunks (one hundred
>  entries?). One single metadata file per entry looks best to me, and is
>  already implemented here:
>
>  http://dev.laptop.org/git?p=users/krstic/ds-backup;a=summary

Ah, ok. I must have mis-read it, I thought it was exporting to a
single large metadata file.

>  >    - Initially I thought that transferring the
>
>  Something missing here?

Nothing important :-) sorry!

>  >  Full restore:
>  >
>  >  We can simplify things here
>  >   - If the XS is using rsync, the REST API returns a path, the XO
>  >  rsyncs from there
>  >   - If the XS is using git internally, there are more options it may
>  >  force a delay (to get a temp checkout that will be held there for
>  >  24hs) or perhaps the client can just execute git archive and pipe the
>  >  output to itself ;-)
>  >   - After retrieval of files from the XS, the XO must rebuild its
>  >  Xapian metadata store
>
>  Should this happen after each entry has been copied?

That would be _very_ costly on the server side. Let the rsync
complete, and do a bulk import.


>  >  Single-item-restore:
>  >
>  >  I'll punt on this in the short-term (as I have to prepare an XS
>  >  release too!), but various avenues are open. We can support web-based
>  >  zipfile download, specially since Tomeu's implemented the handling,
>  >  and we can support Journal-based browse-and-restore. But want to get
>  >  the major issues sorted first ;-)
>
>  Sure. Robson may be interested in lending a hand here. At any point,
>  thinking now about the different possibilities may not do any harm.

Definitely. Right now I am trying to sort out the plumbing that does
the real job, and do the web-based stuff later ;-)

cheers,



m
-- 
 martin.langhoff at gmail.com
 martin at laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff