[sugar] Education?

[Neal Norwitz]

On 3/11/07, Antoine van Gelder <hummingbird at hivemind.net> wrote:
>
> > If you meant byte code, the final stage of the compiler should be able
> > to do that just fine.
>
> Aha! Was not aware that this was already possible! Best to start digging
> in compiler.pycodegen ?

You are so brave!

> Spent the weekend starting to play with a little toy visual scripting
> editor that takes an AST generated from a python source file by
> compiler.parse() as input and am having some wonderings about the best
> way to go from the edited AST to:
>
>    a) byte code for execution
>    b) back to source code for non-visual editing
>
> Of course if I have b) I could simply recompile but I think it would be
> neat to not go the long way round :-)

a) is pycodegen.  This converts the AST that is output from the
compiler package into executable Python byte code.  There are a few
bugs (with the 'with' statement IIRC).  There is a patch, to fix this
problem, but I haven't verified it worked.  (Hmm, I can't find it so
maybe it was applied?)

b) doesn't exist AFAIK, except in proprietary tools (like decompyle).

There's a bit of history here though.  The compiler package was added
in Python 2.0 IIRC.  Internally Python did not use an AST.  The
compiler package is quite slow.  In Python 2.5, an AST was added.
This AST can be exported.  Since it's implemented in C, it's much
faster.  However, there's no relationship to the AST python uses
internally and the compiler package.  It would be nice to hook them up
and replace a lot of the Python code.  (Actually, it would be nice to
be able to use either the pure Python AST or the one generated from
CPython.)

Both ASTs lose comments IIRC.  The Python 2.5 AST has both line
numbers and column numbers.  The AST from the compiler package has
only line numbers.  Either way you would need to read the source to
reconstruct the comments.  Docstrings should be available from either
AST.  If not, Python keeps the docstrings around, so you would be able
to add those back in.

My memory is suspect of course, so you should verify all the details.
:-)  But the basics should give you a better idea what's going on.

We've talked about making a common AST used by CPython, IronPython,
PyPy, and Jython.  However, no work has been done to date and no one
is actively working on it.

n