In summary<br><br> - /srv is pretty full<br><br> - Fedora 7/9 mirrors are large and growing. Perhaps F-7 is no longer needed. Perhaps both mirrors can be moved elsewhere, or dropped. A mod_rewrite redirect can do its magic, pointing to the new location or to an official Fedora mirror.<br>
<br> - The XO image builder runs constantly, sometimes 2 or 3 times a day, and to consume 1.6GB on each run. This seems to be unbounded, so it'll eat all disk, and on disk full it sits there at 90% CPU usage. Not a pretty sight. I wrote a sample garbage collector, but it's not enough -- the missing bit would be a pacemaker that prevents too many builds from happening on a given day (in case the build aborts early, and retries immediately). <br>
<br>the frustration in the messages below comes from the fact that all non-XS tasks were supposed to move away from xs-dev in... June 2008! Fancy new hw came and went, seasons changed... but no news on xs-dev. :-(<br><br>
My needs for xs-dev are pretty simple. A place to host ISOs and a small yum repo for the XS-specific packages. The XO image builder, Fedora mirrors and all that jazz... can be rm'd anytime if you ask me :-)<br><br>[ no! just kidding! the _builder_ needs a new home, or at least some restrictions to prevent it from messing up the whole server - something like <a href="http://littlehedgehog.co.nz/images/catalogue/touchwood-meadow-bar-cot.jpg">http://littlehedgehog.co.nz/images/catalogue/touchwood-meadow-bar-cot.jpg</a> ]<br>
<br>cheers, martin<br><br><div class="gmail_quote"><span style="font-size: large; font-weight: bold;">Forwarded conversation</span><br>Subject: <b class="gmail_sendername">xs-dev disk full?</b><br>------------------------<br>
<br><span class="undefined"><font color="#000000">From: <b class="undefined">Martin Langhoff</b> <span dir="ltr"><<a href="mailto:martin.langhoff@gmail.com">martin.langhoff@gmail.com</a>></span><br>Date: Fri, Feb 27, 2009 at 2:32 PM<br>
To: OLPC Devel <<a href="mailto:devel@lists.laptop.org">devel@lists.laptop.org</a>><br>Cc: XS Devel <<a href="mailto:server-devel@lists.laptop.org">server-devel@lists.laptop.org</a>><br></font><br></span><br>/srv is full on xsdev. I'll do some temporary shuffling around of my<br>
stuff to clear up enough room to do what I need to do...<br>
<br>
... whoever looks after the build scripts that hog it (is it joyride<br>
builds?) better do some decent programming to handle the on-disk<br>
footprint better _or_ some storage planning. We hit disk-full every<br>
month on xs-dev for a task that is not xs related... :-/<br>
<br>
cheers,<br>
<br>
<br>
<br>
m<br>
<font color="#888888">--<br>
<a href="mailto:martin.langhoff@gmail.com">martin.langhoff@gmail.com</a><br>
<a href="mailto:martin@laptop.org">martin@laptop.org</a> -- School Server Architect<br>
- ask interesting questions<br>
- don't get distracted with shiny stuff - working code first<br>
- <a href="http://wiki.laptop.org/go/User:Martinlanghoff" target="_blank">http://wiki.laptop.org/go/User:Martinlanghoff</a><br>
</font><br>----------<br><span class="undefined"><font color="#000000">From: <b class="undefined">Martin Langhoff</b> <span dir="ltr"><<a href="mailto:martin.langhoff@gmail.com">martin.langhoff@gmail.com</a>></span><br>
Date: Fri, Feb 27, 2009 at 2:48 PM<br>To: OLPC Devel <<a href="mailto:devel@lists.laptop.org">devel@lists.laptop.org</a>><br>Cc: XS Devel <<a href="mailto:server-devel@lists.laptop.org">server-devel@lists.laptop.org</a>><br>
</font><br></span><br>So, found a crontab entry hiding in root's entry, commented out so no<br>
more joyride for the time being.<br>
<br>
I'm not sure about the hw on that box before that script gets enabled<br>
again, we need a partition for it to play so it can DoS itself, and<br>
nothing else.<br>
<br>
- Are there free LVMs? Can we pop in a disk or disk pair?<br>
- Does anyone know what tmp dirs the script uses?<br>
- Can we switch it away from root? Then we get back our 5% buffer<br>
space so root can actually fix the problem.<br>
- Please move it to /etc/cron.d - this is an important task of the<br>
server, not a personal "zip my mail" task...<br>
<div><div></div></div><br>----------<br><span class="undefined"><font color="#000000">From: <b class="undefined">Martin Langhoff</b> <span dir="ltr"><<a href="mailto:martin.langhoff@gmail.com">martin.langhoff@gmail.com</a>></span><br>
Date: Fri, Feb 27, 2009 at 3:24 PM<br>To: OLPC Devel <<a href="mailto:devel@lists.laptop.org">devel@lists.laptop.org</a>><br>Cc: XS Devel <<a href="mailto:server-devel@lists.laptop.org">server-devel@lists.laptop.org</a>><br>
</font><br></span><br>Right, so I've<br>
<br>
- moved some silly logs (do we need them?) from<br>
/srv/library/mirror/log to /var/tmp/mirror-logs<br>
- compressed the logs of the joyride builds<br>
<br>
this has freed up some 345MB and I can work with that. A little bit at least.<br>
<br>
Now, when I say<br>
I mean that it's crazy to have these builders cranking out 1.6GB of<br>
stuff every few hs and not have a garbage collector. So I wrote one.<br>
It's not set to run, it's not even +x, I'm not part of the VIG or<br>
anything, so I throw it in as a sample implementation. Feel free to<br>
use it but responsibility is strictly yours<br>
<br>
cat /root/samplecleanup.sh<br>
#!/bin/bash -x<br>
# suggested cleanup - <a href="mailto:martin@laptop.org">martin@laptop.org</a><br>
#<br>
# each joyride build takes 1.6 GB<br>
# let's say we can store 200 of them<br>
<br>
MAXBUILDS=200<br>
BUILDSONDISK=`find /srv/builds/xo-1/streams/joyride -maxdepth 1<br>
-mindepth 1 -type d -name 'build*' | wc -l`<br>
<br>
EXCESS=$(($BUILDSONDISK-$MAXBUILDS))<br>
<br>
if ( $EXCESS -lt 0 ); then<br>
# no fat<br>
exit 0<br>
fi<br>
<br>
for B in `find /srv/builds/xo-1/streams/joyride -maxdepth 1 -mindepth<br>
1 -type d -name 'build*' | sort | head -n $EXCESS`; do<br>
echo "Going to rm $B, mbwaha ha ha ha"<br>
## rm -fr "$B"<br>
done<br>
<br>
You'll want to account for the other streams as well. It'd be trivial<br>
to set a quota for each stream.<br>
<div><div></div></div><br>----------<br><span class="undefined"><font color="#000000">From: <b class="undefined">Martin Langhoff</b> <span dir="ltr"><<a href="mailto:martin.langhoff@gmail.com">martin.langhoff@gmail.com</a>></span><br>
Date: Sat, Feb 28, 2009 at 12:44 AM<br>To: OLPC Devel <<a href="mailto:devel@lists.laptop.org">devel@lists.laptop.org</a>><br>Cc: XS Devel <<a href="mailto:server-devel@lists.laptop.org">server-devel@lists.laptop.org</a>><br>
</font><br></span><br>Also - ntp is installed but not running. The time was off by 15 minute<br>
or so, I ran ntpdate to sync it to the world.<br>
<div><div></div></div><br></div><br><br clear="all"><br>-- <br> <a href="mailto:martin.langhoff@gmail.com">martin.langhoff@gmail.com</a><br> <a href="mailto:martin@laptop.org">martin@laptop.org</a> -- School Server Architect<br>
- ask interesting questions<br> - don't get distracted with shiny stuff - working code first<br> - <a href="http://wiki.laptop.org/go/User:Martinlanghoff">http://wiki.laptop.org/go/User:Martinlanghoff</a><br>