[Server-devel] Serving 400+ students w/ a single central XS - ejabberd nightmare?

[Bryan Berry]

On Sun, 2009-03-08 at 22:36 +1300, Martin Langhoff wrote:
> On Sun, Mar 8, 2009 at 7:52 PM, Bryan Berry <bryan at olenepal.org> wrote:
> > But 200-300 users could be online at once. I think it will be too
> > complicated to tell some of them: "Don't connect to the AP right now,
> > you may overwhelm ejabberd"
> 
> Give ejabberd enough RAM and it won't be a problem. The rest of your
> infrastructure will, however.

I am worried about the XO's and not the XS.

I used the hyperactivity program and got some unscientific results. I
simulated 200 users using the schoolserver constantly. The CPU of the XS
was very busy, staying near 100% usage during the account creation
period and then it leveled off. I didn't monitor it very closely after
that b/c ejabberd is not my chief concern

This dell server has a dual-core Xeon 3.0 GHz processor and 2 GB RAM.
RAM was fine, beam only used 450 MB according to ps_mem.py

sudo ./hyperactivity.py gabble schoolserver.schoolnet.gov.np 200

I ran hyperactivity from my regular dell laptop

My XO showed 200 users in the Network View and became very sluggish. Top
showed that 30-40% of CPU was taken up w/ handling the sugar-shell, the
program which manages the Network View. The dbusdaemon was using roughly
15-20% of the CPU. I tried to launch one our E-Paath flash activities
and it hung displaying the error message "Unresponsive script". I tried
to launch EToys and it failed as well.


> > The XS is quite a complicated ensemble of software having an XS at every
> > school magnifies the administration work.
> 
> What admin work do you foresee on the XS?

      * Making sure that ejabberd keeps running
      * Making sure that the school doesn't have roof leak directly
        above the XS
      * Making sure the schools doesn't remove the power cord and use it
        for something else
      * Make sure the school doesn't repurpose the UPS for the XS for
        charging cell phones
      * make sure the XS doesn't get zapped by a sudden power surge
      * Making sure it is there 3 months later
      * Make sure that dansguardian, squid, dhcp stay up week after week

We don't have any problem gettting the kids to take care of the XO's but
we have a Hell of a time making sure they take care of the XS

> > Additionally our schools only have about 8 hours of electricity per day.
> > I am concerned about the XS losing power suddenly multiple times per
> > day.
> 
> Good point. I've been building everything with daily "poweroffs" in
> mind and every component should handle it. But haven't field-tested...

Sadly, no one has and this is one of my chief concerns

> > we can provide that (RAM)
> 
> Have you go the 4GB barrier in mind? Past 4GB we get into all sorts of
> problems. You'll need a 64-bit machine, and you'll have to convince me
> or someone else to build a 64-bit XS iso rather than the vanilla
> 32-bit we're using now.
> 
> > We have a relatively low latency connection b/w the schools and the XS.
> 
> Low latency, high bandwidth and everyone in the same netblock? No
> routers in the middle. That's what the XS assumes, in any case.
> 
> > the conclusion that Nepal has different requirements than some of the
> > other pilot schools.
> 
> Everybody is a little bit special. By deviating from how the XS is
> designed to be deployed, you will add mgmt work to workaround whatever
> gotchas.




-- 
Bryan W. Berry
Technology Director
OLE Nepal, http://www.olenepal.org