Python memory usage (Was: The "iGoogle bug")

[Bernardo Innocenti]

Ivan Krstić wrote:

> It's how memory is normally managed for Python modules that's  
> problematic; security has little to do with it. There are some  
> efforts around making Python's memory management more friendly to  
> embedded(ish) platforms, and we need to see if we stand to benefit  
> from this kind of work.

I also looked into it, and after some research I still think we could
get back some memory by mmap()'ing bytecode, but not much.

Many modules we import are  just bindings, which have a tendency to
contain a myriad of very short functions.  A lot of memory apparently
goes in the metadata overhead to describe function and class definitions.

Finally, I've been explained that some smarter bindings employ a lazy
technique to greatly reduce their memory usage, on the basis that
applications typically only ever use a small set of functions and
classes from the APIs they bind to.

PyQt works like this, and it's a huge win because Qt contains hundreds
of classes.  I hear the same trick was attempted with PyGTK, but it
required binary incompatible (?) changes and eventually the work halted.

PyGTK is responsible for a huge 0.9 seconds of load time and probably
quite a lot of much memory.  So I guess it would be an interesting
optimization target.

-- 
   //  Bernardo Innocenti - http://www.codewiz.org/
 \X/ One Laptop Per Child - http://www.laptop.org/