[Server-devel] XO's get bogged down w/ chatter (was Re: Serving 400+ students w/ a single central XS )

[Bryan Berry]

I just want to reiterate the following because I have gotten several
e-mails suggesting solutions to scaling the XS. 

The Dell server handles this problem fine. I expect the MSI wind PC will
handle it fine as well.

The problem is w/ the ___XO's___ NOT the servers. It is hard for the
XO's to keep track of all the conversations happening

The problem w/ centralization is creating a huge giant shared chatroom.
It creates a ton of "chatter" that the XO's have to keep track of.

I need to find the upper limit of how many XO's can share one roster
w/out getting significantly bogged down.

On Sun, 2009-03-08 at 16:57 +0545, Bryan Berry wrote:
> On Sun, 2009-03-08 at 22:36 +1300, Martin Langhoff wrote:
> > On Sun, Mar 8, 2009 at 7:52 PM, Bryan Berry <bryan at olenepal.org> wrote:
> > > But 200-300 users could be online at once. I think it will be too
> > > complicated to tell some of them: "Don't connect to the AP right now,
> > > you may overwhelm ejabberd"
> > 
> > Give ejabberd enough RAM and it won't be a problem. The rest of your
> > infrastructure will, however.
> 
> I am worried about the XO's and not the XS.
> 
> I used the hyperactivity program and got some unscientific results. I
> simulated 200 users using the schoolserver constantly. The CPU of the XS
> was very busy, staying near 100% usage during the account creation
> period and then it leveled off. I didn't monitor it very closely after
> that b/c ejabberd is not my chief concern
> 
> This dell server has a dual-core Xeon 3.0 GHz processor and 2 GB RAM.
> RAM was fine, beam only used 450 MB according to ps_mem.py
> 
> sudo ./hyperactivity.py gabble schoolserver.schoolnet.gov.np 200
> 
> I ran hyperactivity from my regular dell laptop
> 
> My XO showed 200 users in the Network View and became very sluggish. Top
> showed that 30-40% of CPU was taken up w/ handling the sugar-shell, the
> program which manages the Network View. The dbusdaemon was using roughly
> 15-20% of the CPU. I tried to launch one our E-Paath flash activities
> and it hung displaying the error message "Unresponsive script". I tried
> to launch EToys and it failed as well.
> 
> 
> > > The XS is quite a complicated ensemble of software having an XS at every
> > > school magnifies the administration work.
> > 
> > What admin work do you foresee on the XS?
> 
>       * Making sure that ejabberd keeps running
>       * Making sure that the school doesn't have roof leak directly
>         above the XS
>       * Making sure the schools doesn't remove the power cord and use it
>         for something else
>       * Make sure the school doesn't repurpose the UPS for the XS for
>         charging cell phones
>       * make sure the XS doesn't get zapped by a sudden power surge
>       * Making sure it is there 3 months later
>       * Make sure that dansguardian, squid, dhcp stay up week after week
> 
> We don't have any problem gettting the kids to take care of the XO's but
> we have a Hell of a time making sure they take care of the XS
> 
> > > Additionally our schools only have about 8 hours of electricity per day.
> > > I am concerned about the XS losing power suddenly multiple times per
> > > day.
> > 
> > Good point. I've been building everything with daily "poweroffs" in
> > mind and every component should handle it. But haven't field-tested...
> 
> Sadly, no one has and this is one of my chief concerns
> 
> > > we can provide that (RAM)
> > 
> > Have you go the 4GB barrier in mind? Past 4GB we get into all sorts of
> > problems. You'll need a 64-bit machine, and you'll have to convince me
> > or someone else to build a 64-bit XS iso rather than the vanilla
> > 32-bit we're using now.
> > 
> > > We have a relatively low latency connection b/w the schools and the XS.
> > 
> > Low latency, high bandwidth and everyone in the same netblock? No
> > routers in the middle. That's what the XS assumes, in any case.
> > 
> > > the conclusion that Nepal has different requirements than some of the
> > > other pilot schools.
> > 
> > Everybody is a little bit special. By deviating from how the XS is
> > designed to be deployed, you will add mgmt work to workaround whatever
> > gotchas.
> 
> 
> 
> 
-- 
Bryan W. Berry
Technology Director
OLE Nepal, http://www.olenepal.org