Regular expressions and the global interpreter lock

[Duncan Grisby]

Hi,

I have encountered a problem with the re module. I have a
multi-threaded program that does lots of regular expression searching,
with some relatively complex regular expressions. Occasionally, events
can conspire to mean that the re search takes minutes. That's bad
enough in and of itself, but the real problem is that the re engine
does not release the interpreter lock while it is running. All the
other threads are therefore blocked for the entire time it takes to do
the regular expression search.

Is there any fundamental reason why the re module cannot release the
interpreter lock, for at least some of the time it is running? The
ideal situation for me would be if it could do most of its work with
the lock released, since the software is running on a multi processor
machine that could productively do other work while the re is being
processed. Failing that, could it at least periodically release the
lock to give other threads a chance to run?

A quick look at the code in _sre.c suggests that for most of the time,
no Python objects are being manipulated, so the interpreter lock could
be released. Has anyone tried to do that?

Thanks,

Duncan.