Mini-Conference Proposal: olpcfs

[Greg Smith (gregmsmi)]

Hi Martin, Scott et al,

I gave the draft a quick read and it looks like a great design,
especially if its doable with minimum development time.

However, changing the File System, especially at the inode level can
have many unforeseen implications. I suggest building a test plan in
advance and doing a thorough design review at functional level and code
level before committing any changes.

It would also help to have a more abstract definition of goals.

How does this help users, administrators or systems integrators?

I think the main goals are:

1 - Allow people to manipulate files at the CLI using the file name
visible in the Journal. E.g. see a file in the journal and then run cp,
ls, grep, etc. on it at the prompt.

2 - Allow activities and Sugar in general to directly access files with
the journal/human readable name. That is, open the activity first then
open the file using the name without going back to open it directly from
the journal.

3 - Speed up access and open time for files from Sugar GUI (including
activities)

That's my guess let me know if its close.

If #1 is a goal, my first question is if all Linux commands will work
with the new FS as is with no modification. We definitely do not have
capacity to re-code all linux commands!

Based on goals above a couple of use cases will help evaluate if the
design is on target.

E.g. 
User wants to see all the files they have created since date xx/yy. Can
they run ls -al on the appropriate directory (e.g. home directory) and
see file names and dates in human readable form?

User wants to copy files to a USB drive. Can they run cp file.name
[mount point]/file.name ?

Activity developer wants their activity to open and save files using
file names. Do they need to include additional code to do that? How hard
is that and will it slow down the addition of new activities? Will all
existing activities run as is with no modification (AKA backward
compatible)?

In short, build a definition of who needs this change and why.

BTW I think that Squid hashes file names and indexes them on disk in a
way that hides the file name. Its designed to improve performance on
searching, track additional file info, and reduce file access time.
AFAIK they never implemented a way to access the files directly outside
the Squid application but seems like a related challenge.

HTHs. 

An abstract user/administrator level explanation of the goals will help
more people comment on the efficacy of the solution.

Thanks,

Greg S

 
----------------------------------------------------------------------

Message: 1
Date: Sun, 23 Mar 2008 19:31:58 -0400
From: "Martin Langhoff" <martin.langhoff at gmail.com>
Subject: Re: Mini-Conference Proposal: olpcfs
To: "C. Scott Ananian" <cscott at laptop.org>
Cc: OLPC Developer's List <devel at laptop.org>
Message-ID:
	<46a038f90803231631p380f5140s4dd79cf74ad96ad5 at mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

On Fri, Mar 21, 2008 at 12:53 PM, C. Scott Ananian <cscott at laptop.org>
wrote:
>  More information: (draft) http://wiki.laptop.org/go/Olpcfs

The thinking behind this is *excellent*.

If we make a couple of minor tweaks and we take that as the description
of the goals, then we can implement that, and a ton more in a couple of
days by using real files in the filesystem, a single file containing the
"fancy" metadata, and committing things into GIT.
Which gets us to the place where we want to be, with 100th the
engineering effort.

The API gets simplified to:

 - a convention of the fact that for each "document" you are given a
directory - which will get shown as a single item in the UI, a la OSX
with .app bundles

 - in that directory, an optional metadata.yaml (or .xml or whatever)
holds additional metadata.

 - the application can make a "save snapshot" call to trigger a commit
(mapped to the "save document") but this is optional because...

 - when the app exits with a 0 the files get committed

 - when the app exits >0 the files get committed in a specially tagged
commit that indicates abnormal termination - so the journal may later
offer to attempt to open that set of files

 - we can use a readonly FUSE view of the repo or just plain git from
journal to retrieve old version -- but any file retrieved to work on
gets extracted to the real FS

That's the "pragmatic pseudo-API". Some technical notes...

 - by virtue of always working with a real FS, we can direct app
developers to use mmap and similar advanced tricks to work efficiently
on data. Supporting mmap is hell if we do our own FUSE implementation,
and we _need_ performance tricks like mmap.

 - by virtue of using a normal FS, it works with tar, zip, rsync,etc.
Every OS that has extended the POSIX basics has had to come up with its
own archiver implementation to handle the extensions (hi Darwin!)

 - the work of the School server with backups is _trivial_ ;-) -- git
push

 - it is trivial to implement a "sliding window" to "forget" old commits
as the disk gets full, and a longer sliding window on the XS

 - we can tweak our git configuration to work well with the Flash and
jj2fs filesystem

A few advanced tricks will need some tweaking (handling low disk
conditions until we can consolidate packs, packing opportunistically
when we have a power source), but by avoiding the FUSE path we can save
a ton of time which we can use for those purposes.

And then, with the experience of doing this the cheap way, we can
schedule a year or two to write a new FS, and have a go at nailing a
better implementation than Torvalds and his ragtag band of kernel
developers ;-)

cheers,



martin
--
 martin.langhoff at gmail.com
 martin at laptop.org -- School Server Architect
 - ask interesting questions
 - don't get distracted with shiny stuff - working code first
 - http://wiki.laptop.org/go/User:Martinlanghoff