[Server-devel] RW Compressed FUSE FSs? (Re: XS - XO archiving and backup)

[Deepak Saxena]

On Nov 10 2008, at 11:08, Martin Langhoff was caught saying:
> On Mon, Nov 10, 2008 at 10:49 AM, Greg Smith <gregsmitholpc at gmail.com> wrote:
> > On backup and restore, aside from the comments already mentioned, I
> > suggest you pay careful attention to the available space on your XS. You
> > should have about 2GB free space on your XS for each XO. If you don't
> > have enough space it may move some of the files over your wireless LAN
> > then not save them on the XS.
> 
> Excellent point. Which leads to a request - good references on a well
> behaved compressed FUSE FS that
> 
>  - supports RW
>  - behaves well with rsync (which I presume mmaps files liberally)
>  - supports hardlinks and ACLs
> 
> As a happy user of large hard drives, I haven't needed a compressed FS
> since the unhappy days of DRDOS so I'm rusty on this front. There's a
> listing at http://fuse.sourceforge.net/wiki/index.php/CompressedFileSystems
> but I know nothing about the quality, reliability and performance.

I took a look at these and in summary, only the following options seem 
useable for the XS scenario as the remainder are for cramfs, loopback,
read-only or other non-general purpose use.

compFUSEd

Not 100% of status. Last update was May 1, 2007.

LZOlayerFS 
 
Has not been updated since Feb 2006 so I say that is an immediate NO
IMHO unless we want to take over the project.

fuseCompress

Looks to be in active development. Last Git update was 11/02 and there
is a steady stream of updates for the last two months. 

> I care mainly about the reliability -- but it better be fast too.
> Compression ratio is perhaps more negotiable, the other two arent :-)

I think the next step in determing which (if any) of the above would meet
the XS needs is to start pounding on them with LTP, iozone and other 
performance/stress/compliance tests to see how well they all work.
Beyond performance, I'm worried about stability/maintainability. Regular
filesytems get a lot of testing by a whole lot of people and when a major
bug hits, we know it will be fixed ASAP. I don't know if the same can 
be said of externally maintained filesystems. I suggest we also 
engage with the developers of the three alternatives above and let 
them know that we're considering deploying them on the XS. 

Note that there is also an ext3 compression extension that is
probably worth investigating, though this requires custom kernel
packaging.

That being said, I think we also need to get a really good idea of
just how much data is already compressed. (We need same data to 
determine whether to enable or disable compression on /home/olpc on 
the XO itself). Running a FUSE layer will defintely result in performance 
degradation (which also means increased power consumption) and to go 
through the whole process of compressing blocks to discard the results 
for the majority of blocks would be a resource waste.

~Deepak

-- 
 Deepak Saxena - Kernel Developer, One Laptop Per Child
   _____   __o                                   (o>
------    -\<,  Give One Laptop, Get One Laptop  //\
 ----- ( )/ ( )  http://www.amazon.com/xo        V_/_
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~