LZO support

[Erik Garrison]

On Tue, Jul 15, 2008 at 04:41:35AM -0400, Greg Smith wrote:
> Hi Erik,
> 
> Can you design a test case or two to test the performance of these 
> compression schemes?

Yes.  The following script encodes a trivial and necessarily flawed test
case:  

    http://dev.laptop.org/~erik/code/testcompression.py

To run this test script you will need http://www.zlib.net/zpipe.c, a
simple zlib utilization which compresses or decompresses stdin to
stdout.  (compile with gcc zpipe.c -o zpipe -lz) somewhere in your path.
Additionally, you will need to install lzop there are packages on both
ubuntu and fedora (yum install lzop; sudo aptitude install lzop).

The script runs compression and decompression of a target file (both to
/dev/null to mitigate I/O bounding) some number of times, averages the
results, and then prints the ratios between each algorithm's compression
ratio, compression time, and decompression time.

The idea is to get a rough estimate of the potential performance
benefits of one algorithm over the other by utilizing them both in the
same environment.

    % python testcompression.py --help
    Usage: testcompression.py [options]

    Options:
      -h, --help    show this help message and exit
      -c CYCLES, --cycles=CYCLES
                    run each compression or decompression CYCLES times


A simple test (on the uncompressed joyride usb image os2163.usb) is as
follows:

    % wget
    http://xs-dev.laptop.org/~cscott/xo-1/streams/joyride/build2163/devel_jffs2/os2163.usb
    % python testcompression.py -c 5 os2163.usb
    running 5 cycles
    testing on os2163.usb
    size =  706715648
    writing lzo compressed file os2163.usb.lzo
    writing zlib compressed file os2163.usb.zlib
    lzo compression .....
    lzo decompression .....
    zlib compression .....
    zlib decompression .....
    results on os2163.usb
    lzo: compression ratio 0.511157, compress 9.281282 s, decompress
    2.848002 s
    zlib: compression ratio 0.401923, compress 51.469934 s, decompress
    4.607102 s
    ratios, lzo : zlib
    --- compression ratio: 1.27177775711
    --- compression time: 0.180324344864
    --- decompression time: 0.618176592161

These results (lzo takes 1/5th the time to compress, 1/2 the time to
decompress, and takes up 120% of the space as zlib) seem to be roughly
in line with my other tests.

Possible flaws:
    - the algorithms use a different blocksize than our fs
    - the algorithms are significantly different from those included in
      the kernel

I would be happy to hear more flaws or ideas about how to improve this
test.

Erik