9.1 Proposal: Top five performance problems

[Mitch Bradley]

Michael Stone wrote:
> I did some basic profiling of my new rainbow code last night and
> discovered that, in the best case with the current codebase on XO, it
> costs about 0.5s/"1 exec(python)". Approximately 80% of the 0.5s was
> spent importing modules.
>
> I hope to dig deeper in the near future, but I am concerned at my lack
> of inspiration about how to deal with this problem. (Other than by
> rewriting into a different language.) I still do not consider the
> mod_python approach used in the 767-era rainbow to be a viable long-term
> solution.
>   

Well, there is a tedious solution that would probably be effective.  Go 
through the list of modules with a fine-toothed comb and find out what 
is actually used from each module.  I'll bet that there are quite a few 
modules from which only a few simple functions are used.  Collecting 
those functions into one lightweight (no unnecessary stuff) module might 
collapse the dependency graph.

As I said, this can be tedious, but it's the sort of think I've done 
many times during my career, and it has usually paid off.  If nothing 
else, you end up learning a lot about how things work, which tends to 
make you eventually become fearless.  Hah! I know how that works, and 
it's not nearly as complicated as you think!

A lot of complexity ends up being solutions to low-value problems that 
don't apply in your case.  As a case in point, a long time ago I needed 
to incorporate a stripped-down stdio package in some app that needed to 
be tiny.  The basic character I/O ended up pulling in a train load of 
networking libraries.  It turned out that "isatty()" was the culprit - 
it had to check whether the file descriptor matched every conceivable 
kind of I/O object.  I just made a stub version of isatty() and all the 
spurious dependencies disappeared.