[Server-devel] Schoolserver development in Uruguay

Daniel Drake dsd at laptop.org
Thu Aug 19 22:56:46 EDT 2010


On 19 August 2010 18:25, Bernie Innocenti <bernie at codewiz.org> wrote:
> == Jabber ==
>
> There are two people working on Jabber. They have been using ejabberd
> and, quite surprisingly, they've not seen any issues of high CPU load
> and database corruption. Tomorrow I'll get to work more with them.

XS-0.6 and some of the package updates that come later fix a few bugs
related to ejabberd CPU/DB. I guess in Paraguay they are still on 0.5.

> This is a black hole in all deployments I visited.
>
> Redundant storage is too expensive. One cheap 500GB hard-drive is
> typical. In one year, 3 of the 10 schoolservers in Caacupé developed a
> hard drive failure.

But it's not a huge issue because the XOs also have a copy of the
journal. So, if technical resources are available for a quick XS
repair, disruption should be minimal.

> Journal backups, however, amount to a whopping 238GB of rapidly
> changing, mostly uncompressible and undeltable data. Quite not the ideal
> case for an incremental backup. With today's available resources, we
> could afford to backup everything *but* the journals.

You're giving numbers but missing an important consideration - the XS
backup system makes multiple backups. And it'll continue to do make
more and more copes until it meets a certain threshold based on disk
size (likely to be 238GB in your case). At this point, it will purge
the oldest backups before making new ones.

Saying that you've hit 238GB after a year isn't conclusive because its
likely that you'll meet the threshold when you're measuring an active
school over such a long time period. It's the design - use the
available space.

It's possible that within that space you have 10 backups of every
journal. So you could possibly get away with a disk half the size, and
"only" retain 5 copies. I'm inventing numbers (and they aren't
strictly copies either), but you can provide real ones - how many
backups (on average) are there of a journal in this server? What's the
disk space used if you only total the space used by the most recent
backup of each journal? Also, is it possible that your space-measuring
script is counting a 5mb file with 2 hardlinks as 10mb of used disk
space?

> Paraguay uses Puppet. We're very happy with it.
> Uruguay uses CFengine. They seem to be very happy with it as well.
>
> Both employ a flat hierarchy with one puppet master controlling all the
> schools, which is simple and straightforward, but requires excellent
> connectivity.

"Excellent" is a bit subjective, but yes, the fact that it requires
any form of connectivity is a roadblock in many cases. However, we
came up with a way around this (ideas only, for now, but wouldn't be
hard to implement) for puppet:
- clone all the puppet repositories and the config files and put them
on a USB disk (and do this periodically)
- install puppet-server on all the XSs (but dont run it by default)
- go to a school with said USB disk, plug it in and run puppet-server
- run puppet-client, connecting to localhost
- stop puppet-server, unplug USB disk, go home

Daniel


More information about the Server-devel mailing list