[sugar] DS possibilities

[Tomeu Vizoso]

On Wed, Apr 23, 2008 at 10:19 PM, Benjamin M. Schwartz
<bmschwar at fas.harvard.edu> wrote:
>  Based on these options, I personally recommend that the priorities be:
>
>  1. Write a new datastore implementation to the same API.
>  1. Set up a datastore test system.
>  2. Improve logging.
>  - --- Future ---
>  3. Write a new datastore implementation with a new API.
>  3. Notify the user when things go wrong.
>
>  My recommendation is not very strong.  I can see many other reasonable
>  arguments.  However, I prefer this one:
>
>  After glancing at Michael's Simple Data Store code, I am convinced that it
>  would be easy to turn it into a complete implementation of the current
>  datastore API, with a backend that is simply individual files in the
>  filesystem.

Where would metadata be stored?

>  Search could be implemented by grep -R, or something of
>  similar complexity.

Hmm, I see filtering and fulltext search as really important features.
Do we really want to grep instead of using some kind of index?

>  I would be perfectly happy to replace the current
>  datastore layer with this one (though the upgrade mechanism will take some
>  careful thought).  If some code from the existing implementation is still
>  applicable, then it may of course be reused.  Indeed, the rewrite vs. fix
>  distinction is something of a false dichotomy; the major issue is whether
>  or not to use Xapian and other advanced algorithmic designs.

I think that most of our problems come from using Xapian for storing
the metadata, but not from using it as what it is intended: a fulltext
search engine. I actually think that it has worked quite well for that
purpose.

>  The principle objection to this path, it seems to me, is the possibility
>  of introducing new (worse) bugs.  A datastore test system would greatly
>  increase confidence in any new implementation.  If the test system can
>  also reproduce crashes in the current implementation, then we may
>  determine with some confidence whether the new implementation is more
>  reliable (under those circumstances).  Logging goes hand-in-hand with
>  testing; it is needed in order to determine the results.  Having better
>  logging in production laptops will be a positive side benefit.

Agreed, but I would like to note that the DS is (to my knowledge) the
component in Sugar with most extensive unit tests.

>  I recognize the importance of providing version control functionality
>  within the datastore, but the deadline for this work appears to be July,
>  pending an August release.  It seems unlikely to me that any
>  future-proofed version control system could be completed, and integrated
>  with the rest of the system, in two months.  If anyone wishes to argue the
>  opposite point, I will be happy to listen, since I desire this feature
>  greatly.  If the version control project is sufficiently important, it may
>  be acceptable to place an expert (presumably Scott) on this full-time
>  targeting a December release.

I think that it depends on which features we wish to add. This scheme
may be enough to provide the user experience that Eben has devised:

- Store a version number as a metadata property. Create a branch every
time that a non-tip revision is resumed. Suggested format: 3.1.1.

- Use xdelta to store backwards deltas between versions of the same
file. Probably calculate a hash of the content so we can optimize out
the no-changes case.

- That's all ;)

See here for the last work from Ben Saller related to versions:

http://dev.laptop.org/git?p=projects/datastore;a=shortlog;h=version_prototype

>  Notifying the user when critical system components stop working is a good
>  goal as long as it doesn't distract from the more important work of making
>  the critical system components work reliably.  It seems to me that the UI
>  team's work on the wide variety of frequent, user-interaction
>  notifications is far more important, and also sets the stage for this work
>  later.

Yes, the "journal full" notification is one example.

Thanks,

Tomeu