very simple datastore reimplementation

[Jameson "Chema" Quinn]

On Fri, May 9, 2008 at 6:43 PM, C. Scott Ananian <cscott at laptop.org> wrote:

> On Fri, May 9, 2008 at 2:26 PM, Joshua N Pritikin <jpritikin at pobox.com>
> wrote:
> > On Fri, May 09, 2008 at 12:10:07PM -0600, Jameson Chema Quinn wrote:
> >> To be more clear about this use case: I think that there should
> definitely
> >> be a way for the onboard datastore to store the metadata for an absent
> file,
> >> with hints about what place(s) to find that file (networked backup, sd
> >> cards, usb devices) and how to recognize it when you do. This should
> include
> >> the possibility for offloading old intermediate versions. Then, even
> when
> >> you do not have access to the backup storage, you can see what you are
> >> missing. This makes the result of suddenly yanking the SD card out more
> >> well-defined (assuming no filesystem corruption), and means you do not
> ever
> >> have to merge/separate two indexes (there is just one index).
> >
> > I was surprised to read this. My opinion is that the index should only
> > include files which are available on local storage. Otherwise the index
> > can fill up with broken links, and it will be difficult to explain why
> > the broken links don't work. Access to backups is a good idea, but not
> > via such a by-default mechanism.
>
> http://wiki.laptop.org/go/Olpcfs#Absent_content_and_merging_remote_stores
>  --scott
>
> --
>  ( http://cscott.net/ )
>

Let me be a little clearer still about what I am envisioning. I do not
imagine that, except in rare cases (restore after total data loss) the
journal would ever scan external/network storage for an absent file, or
explicitly import one or more files (as distinct from opening them or
otherwise doing something meaningful with them, using a traditional
tree/folder view). The journal index would know exactly where things were,
but only if it put them there itself, or had used them there itself. It
would then keep track of whether the containing volume/resource was
available and show the links as broken/real on that basis. If the file had
been deleted foreignly, it might mistakenly show as available but turn out
to be unavailable when clicked on.

This model has several advantages. It does not require any scanning on
mount; it does not add files to the journal if they have never been used;
and its behaviour is generally understandable and predictable. If I am
reading c_scott's link above correctly, it is not precisely what was on his
mind, but it is supported by the same capacities of olpcfs, with the
addition of an index by storage volume and some metadata (indexed only if
you need to search it) for external path(s).

A couple of UI points:
- broken links could be filtered out by default in most views, but it would
be a simple boolean switch (checkbox or the sugar equivalent) in the
interface to show them.

-If UID can hide in the metadata, which, if I understand, is preserved as
part of the file even on foreign (unix-only?) filesystems (wow!), I do not
see any compelling reason for it to be in the filename. Locally-stored files
could have their real filenames, with 2 random characters at a time added in
case of collisions; for external storage, collisions could simply be
forbidden, and you could rename or choose a different directory.

-some file formats already include a compatible metadata spec - for
instance, an MP3 file already specifies artist. a by-format plugin api to
keep the olpcfs metadata in sync with this metadata would be a nice future
feature.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.laptop.org/pipermail/devel/attachments/20080510/eb03a115/attachment.html>