[linux-mm-cc] [PATCH 0/3] compressed in-memory swapping take5

[Nitin Gupta]

John McCabe-Dansted wrote:
> For Ubuntu LiveCDs the difference was dramatic. As I recall, given a
> 256MB test VM, the ubuntu-hardy alpha 4 and Mar 08 daily build would
> crash horribly without compcache. With compcache it ran fine with
> 240MB. And run somewhat slower at 192MB, but still worked in the end
> as I recall.
> 
> Clearly forcing users of 256MB machines (officially a supported
> configuration) to download and use an alternate installer was not
> ideal. They have since added an ubiquity without gnome install.
> Never-the-less, compcache already makes the difference between an
> Ubiquity+gnome install working and not working. If a user happens to
> select "install" from Gnome it is still a good thing that it will work
> rather than crash.
>   https://lists.ubuntu.com/archives/ubuntu-devel/2008-August/026092.html
> 

Thanks for detailed numbers you posted. These will all count next time I
attempt to push it to mainline.

> Compcache vs harddisk may or may not be a win, but any form of swap
> vs. running out of memory is almost always a win. Here is a summary of
> the benchmarks I made for the 500Mhz machine. The numbers after the
> slash are in some sense worst case, as they are the ones when 20 files
> have been opened previously, which could cause stale pages to be left
> in the compcache (for the version I was testing then in anycase).
> Actually all of these numbers are kind of worst case, since a 500Mhz
> Pentium 2/3 machine is rather underpowered by todays standards.
> 

Other worse case scenarios would include cases where we depend lot on
big filesystem caches rather than anonymous memory. Linux kernel compile,
some database workloads come to mind.

> The Fields are: Swap type, number of pdf files, time to open pdf files
> (median of 20 runs)
> 
> CC	10		120.91/121.79
> HDD	10		121.11
> NO	10		121.01
> 
> HDD	15		187.01
> NO	15		187.22
> CC	15		187.99/189.31
> 
> HDD	20		261.91
> CC	20		265.72
> NO	20		637.77
> 
> So for this machine, for this task, there is little difference between
> CC and HDD, but both are much better than NO swap.
> 

500MHz is a real antique piece. Anyways, we can see some positive here:
these slow processors seem to be used in embedded devices like PDAs.
These typically use flash for storage which suffer from wear-leveling
issues. So, this data shows that we can maintain nearly same performance
without resorting to using flash as swap device.

> This data was previously posted to the newsgroup, here are the links again.
> http://www.ucc.asn.au/~mccabedj/compcache/benchmark_new_sh.zip
> The spreadsheet summary is time.ods, the raw data is in final.tgz.
>  http://www.ucc.asn.au/~mccabedj/compcache/time.ods
>  http://www.ucc.asn.au/~mccabedj/compcache/final.tgz
> 

What version of compcache you used to collect this data?
Newer capabilities to forward incompressible data to backing swap disk,
collapsing zero pages, discarding stale data (though slowly) should
help score better in these tests.

> Another worst case thing you could mention with the last version of
> compcache I tested, is that LZO compresses a page of sequential
> integers to 51% of the original size, and xvmalloc requires one page
> for each such "non-good" compressed page. However, even when trying to
> fill memory with sequential integers, they were still 20% good
> compress (presumably from gnome, X etc) so compcache still saved ~20%
> of the swapped out memory. (I have previously sent
> 	regression_test.tar.gz to this newsgroup.) So even if most memory is
> uncompressible standard system processes will still allow compcache to
> reclaim some memory.
> 
> So even in this case, if you had e.g. a embedded device that had 64MB
> of ram but needed to allocate 66MB, compcache could still be of use.
> Under pretty much any circumstance, swapping out to compcache (or HDD)
> will release some memory, and the general principle that if the kernel
> wants swap it should be given swap seems to also apply to compcache
> based swap.
> 


> (Also, I would presume that wkdm would do quite a good job here)
> 

Maybe, not too sure. Anyways, getting WKdm, WK4x4 to mainline is another
pain. It took me 7 review cycles (taking more than a month), IIRC, to get
LZO in mainline.

> Also, they appear to be looking at
>   http://code.google.com/p/compcache/wiki/LTSPPerfSummary
> But the following is much more interesting:
>   http://code.google.com/p/compcache/wiki/LTSPPerf
> 

This data was collected using very old compcache version (thats why I
now marked these pages as deprecated). I would request Nai Xia to recollect
it with 0.5.3.

> Finally there are all the other benchmarks for other versions and
> forms of compcaching (e.g. the patches for 2.4.x). These are useful to
> give an idea as to what compcache 1.0 will be like. Compcache 1.0 will
> presumably be a solid default for everybody, as it will be using
> heuristics to determine the optimal strategy (which may be to switch
> to pure HDD swap).
> 

Auto cache resizing, run-time defragmentation, scalabilty... wow! Its
really a never ending project. Not sure how long I will stay with it
but hope that someone will pick it up and take it forward :)

Thanks,
Nitin