mwlib: reworking re2c files to use ctypes

[schmir at gmail.com]

Martin Langhoff <martin at laptop.org> writes:

> The compiled .so files are a bit of a problem currently for us. We
> ship "activities" (user-installable program bundles) that are usually
> pure python, and (if prepared carefully) can be installed in several
> releases of our OSs, which in turn are based on various Fedora
> releases.
>
> Binaries are not recommended inside of those bundles, but if they link
> to generic libs with stable API/ABI, things are generally ok.
>
> The re2c binaries from mwlib, unfortunately, inteface with Python
> using swig, which means that they end up linking directly to

there's no swig involved, but yes, we interface with the python API.

> libpython. We use Python extensively, so we update somewhat
> aggressively to the latest version in Fedora. So what happens is that
> those SO files end up being tied to specific versions.
>
> There is a different, better way to do this -- to create standalone
> .so files, and to use them from Python using ctypes. That way, we can
> distribute precompiled .so files that are significantly more portable
> (they are still arch and glibc ABI specific).
>
> Would that be of interest to you? Has anyone thought about this, or
> worked on this?

Hi Martin,

I understand the problem. But dynamically loading shared libraries will
only create more problems for us. As soon as Fedora ships python 2.7
the issue will be gone, so you just have to wait a bit.

Sorry, but we have no interest in this code.

>
> I now see that in your latest code you are actually not using
> _expander.re anymore. How does the Python-based tokenizer perform,
> compared to the re2c tokenizer? We care a lot about keeping things
> fast.

the re2c based expander has never been in production as far as I can
remember. I can't comment on any performance differences.

Cheers,
- Ralf