Synchronizing xs-0.3 and xo-??? --- backups

[Eben Eliason]

On Mon, Apr 28, 2008 at 10:14 AM, Tomeu Vizoso <tomeu at tomeuvizoso.net> wrote:
> On Thu, Apr 24, 2008 at 6:18 AM, Martin Langhoff
>  <martin.langhoff at gmail.com> wrote:
>  >
>  >   - Initial HTTP check for protocol and XS availability stays. We drop
>  >  the 'nonce' value.
>  >   - Q: Any reason for this being REST vs XML-RPC for the registration service?
>  >
>  >   - Preparations are simpler
>  >    - No need for a master index. Always backup everything (let rsync
>  >  deal with what needs to be backed up), to a directory that will have
>  >  the last copy we rsync'd. Make extensive use of rsync smarts.
>  >    - We must dump _all_ the metadata to a CJSON file
>  >    - Q: Could we create a single metadata file per document (ahem,
>  >  Journal entry) instead?
>  >    - Q: Does it make sense to always maintain the CJSON-formatted
>  >  metadata anyway? (This would specially make sense if some of the DS
>  >  corruption reports being discussed are tracked down to Xapian.)
>
>  Extracting all the metadata from the index is a relatively expensive
>  process, that could affect very negatively the user experience while
>  it lasts. I really recommend to do it in smaller chunks (one hundred
>  entries?). One single metadata file per entry looks best to me, and is
>  already implemented here:
>
>  http://dev.laptop.org/git?p=users/krstic/ds-backup;a=summary
>
>
>  >    - Initially I thought that transferring the
>
>  Something missing here?
>
>
>  >  Full restore:
>  >
>  >  We can simplify things here
>  >   - If the XS is using rsync, the REST API returns a path, the XO
>  >  rsyncs from there
>  >   - If the XS is using git internally, there are more options it may
>  >  force a delay (to get a temp checkout that will be held there for
>  >  24hs) or perhaps the client can just execute git archive and pipe the
>  >  output to itself ;-)
>  >   - After retrieval of files from the XS, the XO must rebuild its
>  >  Xapian metadata store
>
>  Should this happen after each entry has been copied?
>
>
>  >   - Q: How do we trigger this from Sugar on a just-reflashed machine?
>
>  Which user experience do we want here? Eben?

Well, I guess I need to know a bit more about the technical details
that will be in play in this circumstance.  Since we don't have unique
usernames or passwords, the only identifier for the individual and her
data is her key, right?  How does one obtain her key if, for instance,
 her previous machine was bricked or stolen?  And even if she has it,
must we really require her to type it in?  I suspect we may,
unfortunately.

Once these technical considerations are more clear, we can discuss the
manner in which one initiates a restore.  One clean option is to add a
"Restore from backup" action to the Journal palette, which would
launch a modal dialog asking for any required info and displaying
progress information.  Other questions here include:  Should this be
modal to the Journal?  To the entire OS? Can we allow activities to
run (and save, etc) in the background during a restore?

I'm happy to create a mockup or two for this once I have a clearer
idea of the requirements.

- Eben

>  >  Single-item-restore:
>  >
>  >  I'll punt on this in the short-term (as I have to prepare an XS
>  >  release too!), but various avenues are open. We can support web-based
>  >  zipfile download, specially since Tomeu's implemented the handling,
>  >  and we can support Journal-based browse-and-restore. But want to get
>  >  the major issues sorted first ;-)
>
>  Sure. Robson may be interested in lending a hand here. At any point,
>  thinking now about the different possibilities may not do any harm.
>
>  Thanks,
>
>  Tomeu
>