Synchronizing xs-0.3 and xo-??? --- backups

[Tomeu Vizoso]

On Tue, Apr 29, 2008 at 1:34 AM, Martin Langhoff
<martin.langhoff at gmail.com> wrote:
> On Tue, Apr 29, 2008 at 2:14 AM, Tomeu Vizoso <tomeu at tomeuvizoso.net> wrote:
>  >  >    - Q: Does it make sense to always maintain the CJSON-formatted
>  >  >  metadata anyway? (This would specially make sense if some of the DS
>  >  >  corruption reports being discussed are tracked down to Xapian.)
>  >
>  >  Extracting all the metadata from the index is a relatively expensive
>  >  process, that could affect very negatively the user experience while
>  >  it lasts.
>
>  Is it? Ugh then. (I thought it'd be a cheap op!)

Right now it's slow, but it's as well one of the first things that we
should change in order to improve robustness.

>  > I really recommend to do it in smaller chunks (one hundred
>  >  entries?). One single metadata file per entry looks best to me, and is
>  >  already implemented here:
>  >
>  >  http://dev.laptop.org/git?p=users/krstic/ds-backup;a=summary
>
>  Ah, ok. I must have mis-read it, I thought it was exporting to a
>  single large metadata file.

You are right, sorry. Both things are easily doable.

>  >  >  Full restore:
>  >  >
>  >  >  We can simplify things here
>  >  >   - If the XS is using rsync, the REST API returns a path, the XO
>  >  >  rsyncs from there
>  >  >   - If the XS is using git internally, there are more options it may
>  >  >  force a delay (to get a temp checkout that will be held there for
>  >  >  24hs) or perhaps the client can just execute git archive and pipe the
>  >  >  output to itself ;-)
>  >  >   - After retrieval of files from the XS, the XO must rebuild its
>  >  >  Xapian metadata store
>  >
>  >  Should this happen after each entry has been copied?
>
>  That would be _very_ costly on the server side. Let the rsync
>  complete, and do a bulk import.

Ok, we just have to take in account that the bulk import will block
all activity requiring the DS.

>  >  >  Single-item-restore:
>  >  >
>  >  >  I'll punt on this in the short-term (as I have to prepare an XS
>  >  >  release too!), but various avenues are open. We can support web-based
>  >  >  zipfile download, specially since Tomeu's implemented the handling,
>  >  >  and we can support Journal-based browse-and-restore. But want to get
>  >  >  the major issues sorted first ;-)
>  >
>  >  Sure. Robson may be interested in lending a hand here. At any point,
>  >  thinking now about the different possibilities may not do any harm.
>
>  Definitely. Right now I am trying to sort out the plumbing that does
>  the real job, and do the web-based stuff later ;-)

Cool.

Thanks,

Tomeu