#4081 NORM Never A: previews make datastore queries too slow

[Zarro Boogs per Child]

#4081: previews make datastore queries too slow
-----------------------+----------------------------------------------------
 Reporter:  tomeu      |       Owner:  bcsaller      
     Type:  defect     |      Status:  new           
 Priority:  normal     |   Milestone:  Never Assigned
Component:  datastore  |     Version:                
 Keywords:             |    Verified:  0             
-----------------------+----------------------------------------------------
 After the latest fixes regarding the storage of previews in the datastore,
 performance has improved greatly but is still insufficient.

 As can be seen in the attached kcachegrind profile, retrieving each
 returned entry's data represents 33% of the total execution time.

 If no previews are stored we get barely acceptable speed, and that
 operation is almost negligible.

 Retrieving the data part of a xapian document is so slow because we have
 bundled together all metadata, including the preview. This could be
 avoided in queries because in most cases, queries don't request the
 preview to be returned.

 I can see two ways of solving this:

 - store all properties inside xapian like we do now, but non-binary
 properties as document values and only the preview as document data,

 - store the previews externally, similar to what we do for the actual file
 data. We could have an 'external' data type defined in model.py that, when
 the DS receives that property data through dbus, writes it to disk and
 saves the file path as a property in xapian.

 I think we should do it for FRS if we want previews in the journal with
 acceptable performance.

-- 
Ticket URL: <https://dev.laptop.org/ticket/4081>
One Laptop Per Child <https://dev.laptop.org>
OLPC bug tracking system