[Etoys] [squeak-dev] etoys performance problems

[John M McIntosh]

On Mar 25, 2008, at 10:46 AM, Ties Stuij wrote:

> To start with the known, I've been looking into the project loading
> times, and have managed to cut them in half by removing the gzipping
> on the project files and just using the .pr file in stead of the zip
> which claims to be a .pr. Now this might of course not be the perfect
> solution for all, I did uncover some practices on which I'd like
> either the opinion of Squeak people in general or that of the Etoys
> people in particular.

When I was building the Sophie storage subsystem for OLPC I noticed
the OLPC "hard disk" uses a compressed file system. There is some
technical notes on this suggesting developers do NOT zip files because
this is pointless:  Compress the file, then store on a compressed file  
system.
On read then you use cpu time to uncompress the compressed file, then
uncompress the file.  Waste of CPU time.

As a compromise Sophie documents depending on the platform
contain a zip file which contains uncompressed images and compressed  
text,
or uncompressed images and text if we know the platform target is OLPC.

However I was not sure of the network impact of then passing  
uncompressed
files around.

>
> First, I noticed (together with Luke, who showed me the debugging
> ropes) that in the method asUnzippedStream, on ReadWriteStream, the
> method upToEndWithProgressBar uses '/' an awful lot to give the the
> right approximation to the progress bar. It sucked about 20 percent of
> the total loading time of an activity. So I changed
> upToEndWithProgressBar for upToEnd. Which seems like a more reasonable
> default 'cause I'd guess that usually one won't actually use the
> progress bar functionality.

The progress bars interfere with the actual time needed to perform the
action, we had loading progress bars for Sophie documents but found
for small documents the act of showing the progress bar and updating
it could take 80% of the load time.


>
> But thats not the end of it Etoys-wise. 'Cause it turns out an Etoys
> project file is gzipped twice. First the individual file in
> 'writeForExportWithSources:inDirectory:' and then the containing
> function writeForExportWithSources:inDirectory: as part of the .pr
> bundle. And this for a file that already has lots of it's contents
> compressed, seeing that most of the project file is taken up by jpegs.
> And this would ideally also be the case for sound files, but more on
> that later. The size decrease due to zipping in general is not more
> than 10%. So my question on this topic is: is there an argument for
> retaining the double zipping? And would there be a general need for a
> 'just save an uncompressed .pr file, instead of a bundle'
> option/patch?

See comment above about not zipping JPEGS/image data that is
included in the ZIP file.   We choose to store media (sound/video) in  
a separate
media folder because it was pointless to unpack from a zip file on  
need. However
this might not be possible when you want all the data in the zip file,  
still you could
alter the code so that sound/video files aren't compressed by the zip  
logic when
they are added to the zip file.


In Sophie we also have a ZIP file subclass because we found the ZIP  
storage class
was rather inefficient for ZIP files with many members. For example  
when you lookup
and manipulate a zip file member it scans an ordered list looking for  
the member by name.
We altered that to use a dictionary.

> So my concrete question is: how can one for once and for all delete a
> project instance. I was hoping to at least find some generic
> deleteInstance method in the image, but I couldn't find one (I'm a bit
> of a Squeak newbie).

There are programming tools in the image that allow you to see who is  
holding
onto a project, I'll let someone else comment.

> Sound format
>
> Another strategy to trimming down the image size is using a compressed
> sound format. At the moment we use wav for sound right? If I'm wrong,
> excuse me, but I haven't had time to investigate the matter to
> thourougly. Anyways, since a number of activities have quite a few
> samples in them, ranging from short utterings to long sentences, it
> would be very worth while if we could use a compressed format. From
> the mailinglist I read rumours about Ogg plugin and a gstreamer
> plugin... Is it already possible to use Ogg files in Etoys? If so how?
> I'd already be happy with general directions.

http://tinlizzie.org/olpc/OggPlugin/

I am currently working on a gstreamer plugin for OLPC via funding from  
Viewpoints Research Institute,
and have a target completion date for the end of the month.  This  
plugin will let you build
gstreamer pipeline chains and run them, right now you can build a  
pipleline to decode  audio/video ogg files for
example and send the video to a X11 window and the audio to the  
hardware sound player.

I also hope to have 'sinks' which let you import audio and video into  
Squeak.

--
= 
= 
= 
========================================================================
John M. McIntosh <johnmci at smalltalkconsulting.com>
Corporate Smalltalk Consulting Ltd.  http://www.smalltalkconsulting.com
= 
= 
= 
========================================================================