#4406 HIGH Future : XO leaves trash on USB sticks

[Zarro Boogs per Child]

#4406: XO leaves trash on USB sticks
------------------------+---------------------------------------------------
  Reporter:  gnu        |       Owner:  tomeu                            
      Type:  defect     |      Status:  new                              
  Priority:  high       |   Milestone:  Future Release                   
 Component:  datastore  |     Version:  Development build as of this date
Resolution:             |    Keywords:                                   
  Verified:  0          |    Blocking:                                   
 Blockedby:             |  
------------------------+---------------------------------------------------

Comment(by Eben):

 Replying to [comment:13 tomeu]:
 > Replying to [comment:12 Eben]:
 > > Replying to [comment:11 tomeu]:
 > > > I would also like to add that the issue in my opinion is not xapian
 storing 19 files and two dirs. I think that writing all that info inside
 only one file is equally good or bad for the user.
 > >
 > > I think I agree on this point.  It seems that as long as everything is
 tucked within a single hidden top level directory, things can be equally
 as clean.  In either case, we should attempt to minimize the size of
 anything we write here.
 > >
 > > > The problems with the current approach are:
 > > >
 > > >  * need to scan the whole drive every time,
 > > >
 > > >  * the metadata is stored in a binary format that breaks between
 xapian releases.
 > > >
 > > > The index can be recreated (and cached outside the usb stick), but
 if we store the metadata in the device, using a simpler plain text format
 would be better.
 > >
 > > I'm not sure if I can be of any help regarding the above, from a user
 experience perspective (apart from, of course, recommending we prevent
 things from breaking when possible, but that goes without saying).  If you
 have more specific questions on experience, let me know.
 >
 > Well, the intended user experience impacts directly these questions.
 >
 > If we want to be able to do fulltext search inside usb sticks, then
 we'll need to index the names of the files plus all the metadata we want
 to extract from the files, including part or all of the text inside the
 document. In order to make sure we have the fulltext updated, we'll need
 to scan the device after every mount.

 I guess we'll have to weight the tradeoff between indexing time and the
 convenience of full-text search on external devices.  If scanning takes a
 long time, maybe we should stick to the titles (and metadata?) for now.
 If we store the index locally, I assume that we could be intelligent about
 creating the index such that we don't later rescan files which haven't
 changed since creation of the already cached index.

 > If we want to transfer metadata along with files then we need a way to
 store it. Either we use zip files that encapsulate the file plus the
 metadata or we store it hidden somewhere in the usb stick. The first
 option is less convenient when the files are seen from outside sugar, the
 second is what we have today.

 I think that the zip/bundle approach is the only correct way to do this.
 This "activity closure" is, strictly theoretically, a wholly self-
 contained entity capable of launching as an activity, with the associated
 files, and restoring the associated state and metadata.  (This is what an
 "action entry" in the new Journal represents.)  The difference in
 implementation will likley be that we store a reference to the activity
 that should be used to open it, rather than the actual bundle which would
 waste considerable space, with the goal of implicitly obtaining the
 required activity when such a closure is launched.

 I don't know enough about filesystems and xattr to respond reasonably on
 storing metadata for the ordinary objects/files transferred (those not in
 a "closure").  I think we want to remain consistent with what's available
 on other systems in this regard.  Do these comments give a clear enough
 picture of the intent?

-- 
Ticket URL: <http://dev.laptop.org/ticket/4406#comment:14>
One Laptop Per Child <http://dev.laptop.org>
OLPC bug tracking system