Stability and Memory Pressure in 8.2

[John Gilmore]

When measuring memory usage, "cat /proc/XXX/smaps" provides the most
accurate info available (as far as I know), and produces directly
comparable results in all OLPC software releases.  XXX is the process
number you're examining (first column of "ps" output).

The smaps file also tells you how many of the pages in each allocation
are actually *resident* in memory at this instant; how many are
uniquely used by this process (versus shared with other processes);
and how many of them are *dirty* (written by the process).  It also
includes the info that the "pmap" command produces (from /proc/XXX/maps).

E.g. part of the output for the sugar-shell process includes:

b6094000-b6101000 r-xp 00000000 1f:00 3391       /usr/lib/libcairo.so.2.17.5
Size:                436 kB
Rss:                 244 kB
Pss:                  70 kB
Shared_Clean:        244 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         0 kB
Referenced:          244 kB
b6101000-b6103000 rw-p 0006d000 1f:00 3391       /usr/lib/libcairo.so.2.17.5
Size:                  8 kB
Rss:                   8 kB
Pss:                   8 kB
Shared_Clean:          0 kB
Shared_Dirty:          0 kB
Private_Clean:         0 kB
Private_Dirty:         8 kB
Referenced:            8 kB

This says that there are two parts of the libcairo shared library
mapped into the process's address space.  One is 436 kB and is
readable and executable (it's the code segment).  Of that 436k, 244k
is currently resident in memory (Rss), all of that 244k is shared with
other processes and is clean (not written to).  The second part of
libcairo is only 8k bytes long, it's read/write and not executable
(it's the data segment and the BSS segment), all 8k has been read into
RAM, all 8k is private (not shared with other processes) and dirty
(has been modified by the process).  There's a lot more info in there
too, such as what virtual addresses these things are mapped into, and
from what offset within the file.  The "size", "nm", and "objdump -h"
commands from "yum install binutils" will help you compare these
offsets back into the compiler output and thus into the source code.

Dirty memory is particularly pernicious on the XO, since the XO has
nowhere to keep it except in RAM.  On normal Linux systems, when dirty
pages are not expected to be used again soon, they can be paged out to
the "swap space" on disk.  On the XO, which has no swap space, those
pages burn up RAM permanently, even if the process goes to sleep for a
year and never wakes up again.  The dirty memory is released only when
the process exits (or when the process explicitly unmaps it, which
seldom happens).  These long-term dirty pages produce more memory
pressure (less available memory) for all the other processes that are
actually active and getting work done for the user.

When a Linux system gets memory pressure, it tosses out whatever it
can to swap space (nothing, on the XO) and then it starts throwing
away pages that it knows it can later re-read from the file system.
The resident, clean pages that are mapped from files are what get
thrown away (like the first segment of libcairo above).  When the XO
runs low on memory, this means that it throws away a lot of pages
containing executable code.  If the code on those pages is
subsequently executed, those pages will be read back in from the file
system.  Note that reading in a 4k page from the JFFS2 compressed
filesystem is not a cheap operation; a lot of system CPU time goes
into decompressing it (compared to an ordinary Linux system with a
hard disk and an ext3 filesystem).  Throwing away code pages and then
immediately reading them back in again, over and over, "thrashing",
may be why the XO gets very slow when memory is tight.

It may be possible and useful to store some commonly used executables
and shared libraries as uncompressed files in jffs2, making them much
faster to page back in from Flash.  Nobody has tried doing this, as
far as I know.

I don't know how to instrument the kernel virtual memory subsystem to
gain visibility into which pages are being discarded when, and which
are being read in later.  I think that info would be extremely useful
for debugging 8.2's low-memory hangs.  Thrashing would become obvious
if you see the same pages being read in over and over.  Of course,
when the machine is thrashing, it's hard to see any output from its
kernel...

        John