[Olpc-sysadmin] xs-dev needs urgent attn (Fwd: xs-dev disk full?)

[Martin Langhoff]

In summary

 - /srv is pretty full

 - Fedora 7/9 mirrors are large and growing. Perhaps F-7 is no longer
needed. Perhaps both mirrors can be moved elsewhere, or dropped. A
mod_rewrite redirect can do its magic, pointing to the new location or to an
official Fedora mirror.

 - The XO image builder runs constantly, sometimes 2 or 3 times a day, and
to consume 1.6GB on each run. This seems to be unbounded, so it'll eat all
disk, and on disk full it sits there at 90% CPU usage. Not a pretty sight. I
wrote a sample garbage collector, but it's not enough -- the missing bit
would be a pacemaker that prevents too many builds from happening on a given
day (in case the build aborts early, and retries immediately).

the frustration in the messages below comes from the fact that all non-XS
tasks were supposed to move away from xs-dev in... June 2008! Fancy new hw
came and went, seasons changed... but no news on xs-dev. :-(

My needs for xs-dev are pretty simple. A place to host ISOs and a small yum
repo for the XS-specific packages. The XO image builder, Fedora mirrors and
all that jazz... can be rm'd anytime if you ask me :-)

[ no! just kidding! the _builder_ needs a new home, or at least some
restrictions to prevent it from messing up the whole server - something like
http://littlehedgehog.co.nz/images/catalogue/touchwood-meadow-bar-cot.jpg  ]

cheers, martin

Forwarded conversation
Subject: xs-dev disk full?
------------------------

From: *Martin Langhoff* <martin.langhoff at gmail.com>
Date: Fri, Feb 27, 2009 at 2:32 PM
To: OLPC Devel <devel at lists.laptop.org>
Cc: XS Devel <server-devel at lists.laptop.org>


/srv is full on xsdev. I'll do some temporary shuffling around of my
stuff to clear up enough room to do what I need to do...

... whoever looks after the build scripts that hog it (is it joyride
builds?) better do some decent programming to handle the on-disk
footprint better _or_ some storage planning. We hit disk-full every
month on xs-dev for a task that is not xs related... :-/

cheers,



m
--
 martin.langhoff at gmail.com
 martin at laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff  - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff

----------
From: *Martin Langhoff* <martin.langhoff at gmail.com>
Date: Fri, Feb 27, 2009 at 2:48 PM
To: OLPC Devel <devel at lists.laptop.org>
Cc: XS Devel <server-devel at lists.laptop.org>


So, found a crontab entry hiding in root's entry, commented out so no
more joyride for the time being.

I'm not sure about the hw on that box before that script gets enabled
again, we need a partition for it to play so it can DoS itself, and
nothing else.

 - Are there free LVMs? Can we pop in a disk or disk pair?
 - Does anyone know what tmp dirs the script uses?
 - Can we switch it away from root? Then we get back our 5% buffer
space so root can actually fix the problem.
 - Please move it to /etc/cron.d - this is an important task of the
server, not a personal "zip my mail" task...

----------
From: *Martin Langhoff* <martin.langhoff at gmail.com>
Date: Fri, Feb 27, 2009 at 3:24 PM
To: OLPC Devel <devel at lists.laptop.org>
Cc: XS Devel <server-devel at lists.laptop.org>


Right, so I've

 - moved some silly logs (do we need them?) from
/srv/library/mirror/log to /var/tmp/mirror-logs
 - compressed the logs of the joyride builds

this has freed up some 345MB and I can work with that. A little bit at
least.

Now, when I say
I mean that it's crazy to have these builders cranking out 1.6GB of
stuff every few hs and not have a garbage collector. So I wrote one.
It's not set to run, it's not even +x, I'm not part of the VIG or
anything, so I throw it in as a sample implementation. Feel free to
use it but responsibility is strictly yours

cat /root/samplecleanup.sh
#!/bin/bash -x
# suggested cleanup - martin at laptop.org
#
# each joyride build takes 1.6 GB
# let's say we can store 200 of them

MAXBUILDS=200
BUILDSONDISK=`find /srv/builds/xo-1/streams/joyride  -maxdepth 1
-mindepth 1 -type d -name 'build*' | wc -l`

EXCESS=$(($BUILDSONDISK-$MAXBUILDS))

if ( $EXCESS -lt 0 ); then
   # no fat
   exit 0
fi

for B in `find /srv/builds/xo-1/streams/joyride  -maxdepth 1 -mindepth
1 -type d -name 'build*' | sort | head -n $EXCESS`; do
   echo "Going to rm $B, mbwaha ha ha ha"
   ## rm -fr "$B"
done

You'll want to account for the other streams as well. It'd be trivial
to set a quota for each stream.

----------
From: *Martin Langhoff* <martin.langhoff at gmail.com>
Date: Sat, Feb 28, 2009 at 12:44 AM
To: OLPC Devel <devel at lists.laptop.org>
Cc: XS Devel <server-devel at lists.laptop.org>


Also - ntp is installed but not running. The time was off by 15 minute
or so, I ran ntpdate to sync it to the world.




-- 
martin.langhoff at gmail.com
martin at laptop.org -- School Server Architect
- ask interesting questions
- don't get distracted with shiny stuff  - working code first
- http://wiki.laptop.org/go/User:Martinlanghoff
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.laptop.org/pipermail/olpc-sysadmin/attachments/20090302/898894c3/attachment.htm