python imports and performance (was Re: #5549 ...)

[Tomeu Vizoso]

On Mon, Jun 9, 2008 at 6:35 PM, C. Scott Ananian <cscott at laptop.org> wrote:
> Measuring the performance impact of shuffling imports is tricky.  At
> the moment, all of our modules have imports at the top, so all it
> takes is *one* module with a bad import at the top to basically charge
> everyone the loading cost for all that code.  Therefore, you don't see
> any performance improvement by piecemeal changes: you have to do a
> substantial amount of refactoring before you finally get enough
> imports off the startup path to cause an improvement.  If you remove
> 'import foo' from one place where it's not needed, but leave 'import
> bar', and bar in turn imports foo (whether it's needed or not), you
> don't see any improvement.  But I contend that it is still worth
> doing!  When you finally get around to fixing 'bar', you'll start
> seeing performance improvements.

Yes, but once we have shuffled all the imports in our codebase and
seen little improvement, then we'll need to start patching upstream
code so the imports are inline...

At that point, must be easier to send patches to upstream so that the
expensive initializations happen lazily.

> The first time this issue arose was in conjunction with:
>   http://dev.laptop.org/ticket/5538
> and I definitely *did* see concrete performance improvements from the
> changes I made: but they were in a special case where Pippy wanted to
> be able to 'import sugar.activity' in a reasonable amount of time, but
> *wasn't* doing a full activity startup.  I wasn't benchmarking full
> activity startup because that's not what I was interested in.  The
> hostile reaction to my initial patch did not convince me it would be
> worth my time extending my work to the point where it made a
> difference to full activity startup, which undoubtedly would be a more
> invasive patch.

If we can shuffle imports around and drop the prefork hack, I'd be all
for it. Unfortunately, import shuffling hasn't shown yet as much
performance benefits.

> I think the full solution will probably be a combination of lazy
> module loading and moving the most egregious gratuitous imports.
> 'import sugar.activity.activity' still takes 3.7s *by itself* on build
> 703.  Why?

See below for a tip on how to check.

> p.s. I suspect part of the larger startup issue may be that we are
> rendering SVGs on demand for the toolbar, etc.  We can either defer
> that rendering, allowing us to open the activity quickly and then
> later fill in the icons (as the gnome panel does), or maintain a
> persistent cache of SVG renderings.

I don't remember seeing icon rendering in the activity startup
profile, but is easy to test:

import os
import cProfile
import lsprofcalltree

profiler = cProfile.Profile()
profiler.enable()

### code to profile ###

profiler.disable()

k = lsprofcalltree.KCacheGrind(profiler)
data = open('/tmp/import.kgrind', 'w+')
k.output(data)
data.close()

http://www.gnome.org/~johan/lsprofcalltree.py

You can read the .kgrind file with kcachegrind.

Good luck,

Tomeu