[Server-devel] Backup And Restore Feature Documentation

[Martin Langhoff]

On Sat, Sep 13, 2008 at 2:58 AM, Greg Smith <gregsmitholpc at gmail.com> wrote:
> 1 - Does the backup attempt every 30 minutes and run successfully no
> more than 1 per 24 hours?

yes, with a subtlety. No more than once "per day". If it succeeds
today at 3pm, it'll start trying again tomorrow as soon as you turn it
on. It's "next day" though 24 hs have not yet passed...

> Is this turned on by default in 8.2 and is
> there anyway to turn it off (pretty sure that's a default is on and you
> can't turn it off but wanted to check one last time)?

yes

> What happens if
> there is no school server? What happens if there is a school server
> listed in the networks control panel but its unreachable? What happens
> if you are not registered will it still try to backup (maybe just open
> and close every 30 minutes)?

In all those cases, very low power / mem / cpu cost to check for those
factors... every 30 minutes.

> 2 - Does it backup everything in the journal? I think the answer last
> time I asked was "everything in the datastore" which I believe =
> journal, but wanted to check one last time.

yes

> 3 - What is the quota and can it be changed? What is the maximum
> snapshot age and can it be changed? Also, I don't fully understand what
> "snapshot" means.
> Here's an example:
> day 1 it backs up my journal which has 100 items
> day 2 it backs up again and I have 5 new items.
> day 3 it backs up again and I have 10 more items, but now I exceeded my
> quota so it goes and deletes the old snapshot.

A few notes here...
 - The quota is dynamic - depends on # of registered XOs and size of
disk on the XS...
 - If the planning was right,  the quota should be larger  than the XO
disk storage (there's a gotcha: we don't check for that bit of
sanity...)

Making the assumptuino that the quota on the XS is larger than on the
XO, and going back to your example...

 - day 1 you have 100 items - fits on your XO
 - day 2 - 5 new items - plus 2 changes to existing items; backup
process runs and teh XS has 107 distinct items
 - day 3 comes - add 10 more items, pushed you over the XO storage
capacity, so you *first* delete 3 "old" things...

The backup will run and copy everything -- it's a soft quota, so you
can go over a bit when ithe backup process is running. Once it's all
in the XS, it'll

  - "prune" the "day 1" items that don't appear in day 2 (in this
case, the old version of those 2 items you changed)
  - are we within quota? if not...
  - prune the "day 2" items that don't appear in day 3. Those 3 old
things you deleted...

So the scheme works on the principle that there is a significantly
larger quota on the server. If not, it's much less useful...

Additional notes:

After 90 days... - if the quota is large enough that you can keep >90
days (configurable) then past the 90 days we keep 1 snapshot per
month. So you have day1 to day 90 with per-day granularity and then
you see "March 1st 2008, Feb 1st 2008...". Right now those 'monthly
snapshots' get axed first if we are in a squeeze, but I may improve
the logic later...

(So perhaps don't document too much detail about that ;-) )

How the "per user quota" is defined... We look at the size of the disk
that holds the /library directory, and assume that 70% (configurable)
is for backups. Count the # of users registered. Divide.

As you can imagine, this means that the first days in the life of a
school server, the quota allocation jumps around a bit, and then
shrinks as everyone registers and the number of student stabilises.

TODO:

 - This is not stable over time - with a stable-sized school, kids
finish, and new kids enrol, kids and teachers change schools. So the
disk quota shirks indefinitelly. Figure out (a) how to decide that a
user is really gone and (b) WTH to do then? :-)

 - It can also be a source of a minor DoS attack: create a ton of
bogus registrations to squeeze the quota.

> 4 - If you restore a single item, does it get placed in the journal back
> where it originally came from or is it moved to the top?

I think yes. Tomeu (CC'd) knows this one...

> In either case
> does it overwrite the previous instance if it still exists? Should be OK
> to overwrite as I think they should be the same file, just want to be
> sure I understand.

I think yes... Tomeu?

> I apologize if I asked all this before. I think the last time was before
> the feature was complete. I want to get it right for the documentation.

I think these are all new questions :-)

> Feel free to update this
> http://wiki.laptop.org/go/XS_Blueprints:Datastore_Simple_Backup_and_Restore
> with the answers or you reply to me I'll update it then ask for a final ack.

Will leave it in your hands for the moment if that's ok :-)

> FYI I wasn't always such a documentation pest :-( Somehow that's been my
> role on this list lately, but in the past I was more interested in
> designing workflow and SW.

No prob. I find myself being a bit of release manager on other tracks
and... ouch!

> If you want to see some hardcore documentation see the manual for one of
> my previous products (a CDN)
> Replication Status section seems tangentially related to this XS
> feature:

CDN with cute green-yellow-red semaphores? jg will want to talk to you :-)

cheers,



m
-- 
 martin.langhoff at gmail.com
 martin at laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff