#5744 HIGH Never A: "Resuming" a large file from USB copies it into NAND (filling NAND)

[Zarro Boogs per Child]

#5744: "Resuming" a large file from USB copies it into NAND (filling NAND)
-----------------------+----------------------------------------------------
 Reporter:  gnu        |       Owner:  tomeu         
     Type:  defect     |      Status:  new           
 Priority:  high       |   Milestone:  Never Assigned
Component:  datastore  |     Version:  Build 653     
 Keywords:  security?  |    Verified:  0             
 Blocking:             |   Blockedby:                
-----------------------+----------------------------------------------------
 (From rt# 2968)  User was browsing media from a USB drive, using Watch &
 Listen.  Their internal flash filled up, and it made their system
 unbootable.

 I tried to reproduce it tonight.  Easy.

 I installed Watch&Listen from the wiki Activities page, put a 300+MB mp3
 file on a USB memory stick, inserted it into the XO, went to the journal,
 clicked on the USB logo, found the file (despite the Journal tearing off
 half of its file name), clicked it to get details, hovered over the upper
 right icon, and resumed it with Watch & Listen.  I had less than 300MB
 free in my internal NAND flash.

 Something started copying the file into internal flash.  From previous
 experience (#5719) I looked at the datastore process.  It had 21 file
 descriptors open (in /proc/1618/fd), and 20 and 21 were:

   20 -> /media/COMPUSA/raiders_of_old_california.mp3

   21 -> /home/olpc/.sugar/default/data/raiders_of_old_california.mp3

 The file was copied until the NAND filesystem filled up.  (9 MB free).  A
 truncated copy of the file was left in .sugar/default/data.

 After I closed the "detail" page from the Journal, a second copy of the
 file started to get made in the data/ directory!!!  It didn't get very
 far, since there was minimal space left.

 The Journal was unable to unmount the USB stick, because "umount" failed
 with a file system full error.

 Earlier, I tried the same thing with a 15MB mp3 file.  The file also got
 copied to the datastore.  It was there while the file was played by the
 media player.  It was removed after the media player terminated.

 Then I rebooted and tried the same thing with a 125MB Ogg Theora file from
 the Ubuntu Month of Screencasts
 (http://screencasts.ubuntu.com/videos/20070911_updating_and_upgrading_theora_400k_vorbis_1280x720.ogg).
 The Watch&Listen activity started up, even while the USB light was
 blinking as the file was copied -- but the activity could not access the
 file.  The Play button was useless; the timeline didn't show the length of
 the video file.  Not until the entire file was copied -- more than a
 minute.  Ultimately it never was able to play the file at all -- I don't
 know why.  But TWO copies of it were made in default/data -- not two links
 to the same file, but two separate copies.  I don't know why.  I
 terminated the program, yet the file copies remained!  When I tried to
 resume the same file with EToys, it made a THIRD copy, which filled the
 file system, causing all kinds of mischief including making X crash and
 restart every 20 seconds or so.

 After deleting the files to make the X crashes stop, I then went back and
 tried resuming this file with eToys.  eToys started up, said "Opening
 journal entry
 20070911_updating_and_upgrading_theora_400k_vorbis_1280x720", then shortly
 afterward, said "Error: dbus send timed out".  Apparently it was copying
 the file yet again, and etoys was too impatient to wait.  Yep, even after
 it timed out, the USB access light is flashing, and now another copy is
 appearing.

 There's something deeply broken about how Sugar, the Journal, and the
 Datastore deal with files -- particularly on removable or networked media.

 The current design precludes an activity from working with any file larger
 than the available NAND space (1GB maximum).  No activity is ever going to
 play a DVD off a USB DVD drive.  No activity is going to be able to record
 more than 1GB of video from the onboard video camera, no matter if 300GB
 of hard-drive space is available (either plugged in on USB, or over the
 network in a school server).

 Even if the current policy of copy-everything was defensible, it's
 incredibly inefficient.  The activity can't use the file until it has been
 fully copied (and automatically compressed!).  Then the copy has to be
 deleted.  Both operations are expensive and useless, when all that is
 really needed is an open().

 Instead, like everything else on Linux, activities should be reading the
 file(s) they operate on directly from whatever filesystem they are on --
 whether that's local or remote, removable or not.

-- 
Ticket URL: <http://dev.laptop.org/ticket/5744>
One Laptop Per Child <http://dev.laptop.org>
OLPC bug tracking system