[linux-mm-cc] [PATCH 00/12] Avoid OOM-Killer with Ccache

[IKEDA Munehiro]

Hi all,

Following patches are intended to avoid OOM-Killer with Ccache.

Current Ccache has no functionality to free fs_backed chunks 
unless beeing accessed.  This means compressed page-cache pages 
which are used only once will never be freed from Ccache.  As the 
result of it, not only Ccache is full of useless chunks, but also 
Ccache fails to allocate for decompression and OOM-Killer runs.
This phenomenon is easily reproduced under a condition like
  - mem = 16MB
  - max_fs_backed_ccache = 1024 (== 4MB if PAGE_SIZE==4KB)

by running a script like below.
  #! /bin/sh
  find /var -type f -exec cat {} \; > /dev/null

* Needless to say, whether OOM-Killer will run or not depends on 
  how many/what kind of files in /var.
  The more file there are, the more likely OOM-Killer runs.

To solve this problem, I implemented "eat-ccache-allocator" which 
frees the oldest fs_backed chunks when fails to allocate.  In 
addition to it, I introduced a feature which simply frees chunks 
without decompression if the memory reclaim context is accessing 
it to try to free it.

This patch set which consists of 12 patches has 3 parts.

- 1st part (01/12 - 04/12)
Clean-up part.
These patches don't change any functionalities of Ccache, but 
they are needed to apply follwing patches.

- 2nd part (05/12 - 11/12)
Core functionality part of this patch series.
Features described above are implemented here.
(Last one (11/12) is just for clean-up-after in fact)

- 3rd (12/12)
Only touching the proc entry (/proc/ccache_stats) for convenience.


These patches are for source tree of
linux-2.6.21
+ patch-ccache-alpha-008-2.6.21
+ debug patch I posted on May 9th
  http://lists.laptop.org/pipermail/linux-mm-cc/2007-May/000094.html

Below patch helps you to avoid make error.
(But it's not mandatory)
http://lists.laptop.org/pipermail/linux-mm-cc/2007-May/000093.html


Comments, suggestions, bug reports, evaluation reports, and any 
others are welcome.

Happy hacking!

-------

Below are memorandum about remained work related these patches, 
in my mind.

Simply freeing behavior should be applied code path of process 
exiting same as memory reclaiming path.

Current chunk_head locking mechanism is based on spinlock.
Page locking mechanism in mainline kernel is based on wait-on-bit-
lock.
Should we change the mechanism like mainline kernel?
Probably we should evaluate them and decide which is preferable 
because it may affects performance, IMHO.
(I have a trial implementation already in fact.)


Best regards,
IKEDA, Munehiro