[linux-mm-cc] [PATCH 0/3] compressed in-memory swapping take5

[John McCabe-Dansted]

On Wed, Apr 8, 2009 at 9:58 PM, Nitin Gupta <ngupta at vflare.org> wrote:
> John McCabe-Dansted wrote:
>>
>> On Thu, Apr 2, 2009 at 8:02 AM, Nitin Gupta <nitingupta910 at gmail.com>
>> wrote:
>>>>>
>>>>>  - Links to more performance numbers, use cases can be found at:
>>>>> http://lkml.org/lkml/2009/3/17/116
>>>>
>>>> The sole, whole, entire point of this patchset is performance.  Yet
>>>> after chasing a few scruffy links, the only data we have to justify
>>>> merging _any_ of this stuff is, and I quote,
>>
>> FYI, I disagree with this. AFAICT from the point of view of the Ubuntu
>> project
>> the "sole, whole, entire point" of this patchset is "to lower memory
>> requirements in various [diskless] setups.", e.g. LiveCDs and LTSP.
>>   https://wiki.edubuntu.org/Compcache
>>
>
> Looks like simply a better justification could have landed it in mainline :)

For Ubuntu LiveCDs the difference was dramatic. As I recall, given a
256MB test VM, the ubuntu-hardy alpha 4 and Mar 08 daily build would
crash horribly without compcache. With compcache it ran fine with
240MB. And run somewhat slower at 192MB, but still worked in the end
as I recall.

Clearly forcing users of 256MB machines (officially a supported
configuration) to download and use an alternate installer was not
ideal. They have since added an ubiquity without gnome install.
Never-the-less, compcache already makes the difference between an
Ubiquity+gnome install working and not working. If a user happens to
select "install" from Gnome it is still a good thing that it will work
rather than crash.
  https://lists.ubuntu.com/archives/ubuntu-devel/2008-August/026092.html

Compcache vs harddisk may or may not be a win, but any form of swap
vs. running out of memory is almost always a win. Here is a summary of
the benchmarks I made for the 500Mhz machine. The numbers after the
slash are in some sense worst case, as they are the ones when 20 files
have been opened previously, which could cause stale pages to be left
in the compcache (for the version I was testing then in anycase).
Actually all of these numbers are kind of worst case, since a 500Mhz
Pentium 2/3 machine is rather underpowered by todays standards.

The Fields are: Swap type, number of pdf files, time to open pdf files
(median of 20 runs)

CC	10		120.91/121.79
HDD	10		121.11
NO	10		121.01

HDD	15		187.01
NO	15		187.22
CC	15		187.99/189.31

HDD	20		261.91
CC	20		265.72
NO	20		637.77

So for this machine, for this task, there is little difference between
CC and HDD, but both are much better than NO swap.

This data was previously posted to the newsgroup, here are the links again.
http://www.ucc.asn.au/~mccabedj/compcache/benchmark_new_sh.zip
The spreadsheet summary is time.ods, the raw data is in final.tgz.
 http://www.ucc.asn.au/~mccabedj/compcache/time.ods
 http://www.ucc.asn.au/~mccabedj/compcache/final.tgz

--

Another worst case thing you could mention with the last version of
compcache I tested, is that LZO compresses a page of sequential
integers to 51% of the original size, and xvmalloc requires one page
for each such "non-good" compressed page. However, even when trying to
fill memory with sequential integers, they were still 20% good
compress (presumably from gnome, X etc) so compcache still saved ~20%
of the swapped out memory. (I have previously sent
	regression_test.tar.gz to this newsgroup.) So even if most memory is
uncompressible standard system processes will still allow compcache to
reclaim some memory.

So even in this case, if you had e.g. a embedded device that had 64MB
of ram but needed to allocate 66MB, compcache could still be of use.
Under pretty much any circumstance, swapping out to compcache (or HDD)
will release some memory, and the general principle that if the kernel
wants swap it should be given swap seems to also apply to compcache
based swap.

(Also, I would presume that wkdm would do quite a good job here)

--

Also, they appear to be looking at
  http://code.google.com/p/compcache/wiki/LTSPPerfSummary
But the following is much more interesting:
  http://code.google.com/p/compcache/wiki/LTSPPerf

--

Finally there are all the other benchmarks for other versions and
forms of compcaching (e.g. the patches for 2.4.x). These are useful to
give an idea as to what compcache 1.0 will be like. Compcache 1.0 will
presumably be a solid default for everybody, as it will be using
heuristics to determine the optimal strategy (which may be to switch
to pure HDD swap).

-- 
John C. McCabe-Dansted
PhD Student
University of Western Australia