very simple datastore reimplementation

[Tomeu Vizoso]

Hi Martin,

I would love to see such a simple implementation if possible. Nothing
gives more robustness to a component than dropping it. Some questions
follow below.

On Tue, May 13, 2008 at 8:56 AM, Martin Langhoff
<martin.langhoff at gmail.com> wrote:
> On Sat, May 10, 2008 at 2:15 AM, Jim Gettys <jg at laptop.org> wrote:
>  > FUSE is great, but...
>  >
>  >  It means interoperability must be an explicit planned-in-advance action:
>
>  Exactly. That's also my beef with the FUSE approach. The things we can
>  do via FUSE we can do in the regular FS. So let's avoid FUSE, and use
>  a *simple* convention on how to store things in the FS. Something
>  roughly along the lines of:
>
>   - Make a directory per "document", name it
>    <user-picked-name>-<10-char-hash>

This is one of the issues with "human readable" file names: the user
is not forced to pick a name. The closer we can use is the title
property in the metadata, but that needs to be processed so its safe
to be used as a file name across different filesystem types, and if
the user doesn't set titles, we'll end up with lots of entries with
very similar names (thus only distinguished by their hash). Eben has
plans to suggest more strongly to set a title, so this should get
better soon.

>   - Inside that directory, metadata.txt has simple metadata in a nice,
>  human-readable format.

Fine.

>   - If there are versions for this file, perhaps they are in a .git
>  directory, or similar (.svn anyone? ;-) )

Which API would be used by activities and the journal to access versions?

>   - Otherwise, the files in the directory are whatever the activity or
>  application considers to be "the document"

Fine.

>   - We use this same format on external media and on local media

What if the user adds a metadata property of a plain old file in an
usb stick? The folder structure needs to be created?

>   - Jornal can maintain a hidden "documents metadata" database at the
>  root of each FS. This is merely a cached index to have faster mount
>  times (avoiding a rebuild of said index if we can deem it safe).

Ok.

>   - We need a reasonable failsafe behaviour if we find that
>  metadata.txt is b0rken and unreadable.

Yup.

>  and that's it. It's a trivial thing, will work on any fs, on any OS,
>  no magic tricks needed. We can do fast searches on based on the
>  "documents metadata", and the only "slow" op is mounting a device
>  where the documents metadata is stale or missing.
>
>  IMO, jg is hinting towards something like this. If something like this
>  is done in a way that it uses atomic ops most of the time (using an
>  idiom of "write to tmpfile, close and then mv" instead of
>  write-in-place) then we have something pretty much bulletproof,
>  failsafe, and fast. Or at least as fast as the kernel-level FS
>  implementation is.
>
>  And it's bound to be a small simple thing too :-)

Some more issues:

- What about security?

http://wiki.laptop.org/go/OLPC_Bitfrost#P_DOCUMENT:_file_store_service
http://wiki.laptop.org/go/OLPC_Bitfrost#P_DOCUMENT_RO

- What should be done if an activity stores the entry in an invalid
format? If we only have guidelines, we cannot enforce any rules at
write time.

Thanks,

Tomeu