[Server-devel] Squid tuning recommendations for OLPC School Server tuning...

[Martin Langhoff]

Hi!

I am working on the School Server (aka XS: a Fedora 9 spin, tailored
to run on fairly limited hw), I'm preparing the configuration settings
for it. It's a somewhat new area for me -- I've setup Squid before on
mid-range hardware... but this is... different.

So I'm interested in understanding more aobut the variables affecting
memory footprint and how I can set a _hard limit_ on the wired memory
that squid allocates.

In brief:

 - The workload is relatively "light" - 3K clients is the upper bound.

 - The XS will (in some locations) be hooked to *very* unreliable
power... uncontrolled shutdowns are the norm. Is this ever a problem with Squid?

 - After a bad shutdown, graceful recovery is the most important
aspect. If a few cached items are lost, we can cope...

 - The XS hardware runs many services (mostly webbased), so Squid gets
only a limited slice of memory. To make matters worse, I *really*
don't want the core working set (Squid, Pg, Apache/PHP) to get paged
out. So I am interested in pegging the max memory Squid will take to itself.

 - The XS hw is varied. In small schools it may have 256MB RAM (likely
to be running on XO hardware + usb-connected ext hard-drive).
Medium-to-large schools will have the recommended 1GB RAM and a cheap
SATA disk. A few very large schools will be graced with more RAM (2 or
4GB).

... so RAM allocation for Squid will prob range between 24MB at the
lower-end and 96MB at the 1GB "recommended" RAM.

My main question is: how would you tune Squid 3 so that

 - it does not allocate directly more than 24MB / 96MB? (Assume that
the linux kernel will be smart about mmapped stuff, and aggressive
about caching -- I am talking about the memory Squid will claim to
itself).

 - still gives us good thoughput? :-)



So far Google has turned up very little info, and it seems to be
rather old. What I've found can be summarised as follows:

 - The index is malloc'd, so the number of entries in the index will
be the dominant concern WRT memory footprint.

 - Each index entry takes between 56 bytes and 88 bytes, plus
additional, unspecificed overhead. Is 1KB per entry a reasonable
conservative estimate?

 - Discussions about compressing or hashing the URL in the index are
recurrent - is the uncompressed URL there? That means up to 4KB per
index entry?

 - The index does nto seem to be mmappable or otherwise

We can rely on the (modern) linux kernel doing a fantastic job at
caching disk IO and shedding those cached entries when under memory
pressure, so I am likely to set Squid's own cache to something really
small. Everything I read points to the index being my main concern -
is there a way to limit (a) the total memory the index is allowed to
take or (b) the number of index entries allowed?

Does the above make sense in general? Or am I barking up the wrong tree?


cheers,



martin
-- 
 martin.langhoff at gmail.com
 martin at laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff