PyThreadState_Swap difference in 2.3.2?

[Paul Miller]

I have a C++ application that uses multiple embedded Python interpreters. I
allocate multiple interpreters using Py_NewInterpreter, and switch between
them using PyThreadSate_Swap.

In 2.2.2, this all worked fine.

I just installed 2.3.2, but now the first time I call Py_NewInterpreter, it
bails out in PyThreadState_Swap, in the debug checking code, with:

Py_FatalError("Invalid thread state for this thread");

Has the interpreter/thread setup code changed since 2.2?

To be complete, I set up with this:

Py_Initialize();
PyThreadState *globalState = PyThreadState_Get();

// allocate new interpreter
PyThreadState *interp = Py_NewInterpreter();

I am investigating to see if a crash I used to see with 2.2 is still around
in 2.3, where if you call Py_Initialize() and Py_Finalize() multiple times
it would die.

 

[vincent wehren]

| I have a C++ application that uses multiple embedded Python interpreters.
I
| allocate multiple interpreters using Py_NewInterpreter, and switch between
| them using PyThreadSate_Swap.
|
| In 2.2.2, this all worked fine.
|
| I just installed 2.3.2, but now the first time I call Py_NewInterpreter,
it
| bails out in PyThreadState_Swap, in the debug checking code, with:
|
| Py_FatalError("Invalid thread state for this thread");
|
| Has the interpreter/thread setup code changed since 2.2?
|
| To be complete, I set up with this:
|
| Py_Initialize();
| PyThreadState *globalState = PyThreadState_Get();
|
| // allocate new interpreter
| PyThreadState *interp = Py_NewInterpreter();

The same thing stumped me too a couple of days ago (s.
http://groups.google.nl/[email protected]&rnum=1 )
with 2.3.3c1 (same applies 2.4a0 for that matter).

Although I am sure the following snip in PyThreadState_Swap was put there
for a reason,
it would be great if someone could put some light on the matter....:

/* It should not be possible for more than one thread state
to be used for a thread. Check this the best we can in debug
builds.
*/
#if defined(Py_DEBUG) && defined(WITH_THREAD)
if (new) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check && check != new)
Py_FatalError("Invalid thread state for this thread");
}
#endif

Regards
Vincent Wehren


|
| I am investigating to see if a crash I used to see with 2.2 is still
around
| in 2.3, where if you call Py_Initialize() and Py_Finalize() multiple times
| it would die.
|
|

 

[Paul Miller]

The same thing stumped me too a couple of days ago (s.

http://groups.google.nl/[email protected]&rnum=1 )
with 2.3.3c1 (same applies 2.4a0 for that matter).

Interesting. Unfortunately, there isn't a whole lot of information out
there on what we're trying to do.

I went ahead and commented out that check, but then I came across another
problem. In object.c, it hit an assertion:

assert(mro != NULL);

This was for a custom Python type I had created in C. It was assuming this
mro field was initialized, which it wasn't. It must be something new,
because this all used to work in 2.2

I will have to keep looking at the release notes.

I wish there was a "upgrading from 2.2 to 2.3" document for people who are
writing custom Python extensions, and doing embedding and such.

 

[Mark Hammond]

| I have a C++ application that uses multiple embedded Python interpreters.
I
| allocate multiple interpreters using Py_NewInterpreter, and switch between
| them using PyThreadSate_Swap.
|
| In 2.2.2, this all worked fine.
|
| I just installed 2.3.2, but now the first time I call Py_NewInterpreter,
it
| bails out in PyThreadState_Swap, in the debug checking code, with:
|
| Py_FatalError("Invalid thread state for this thread");
|
| Has the interpreter/thread setup code changed since 2.2?
|
| To be complete, I set up with this:
|
| Py_Initialize();
| PyThreadState *globalState = PyThreadState_Get();
|
| // allocate new interpreter
| PyThreadState *interp = Py_NewInterpreter();

The same thing stumped me too a couple of days ago (s.
http://groups.google.nl/[email protected]&rnum=1 )
with 2.3.3c1 (same applies 2.4a0 for that matter).

Although I am sure the following snip in PyThreadState_Swap was put there
for a reason,
it would be great if someone could put some light on the matter....:

/* It should not be possible for more than one thread state
to be used for a thread. Check this the best we can in debug
builds.
*/
#if defined(Py_DEBUG) && defined(WITH_THREAD)
if (new) {
PyThreadState *check = PyGILState_GetThisThreadState();
if (check && check != new)
Py_FatalError("Invalid thread state for this thread");
}
#endif

Python stores information in the thread-state structure which is
specific to the thread - information such as the exception being
handled, the current recursion depth, etc.

The assertion above is reporting the fact that 2 distinct ThreadState
objects are trying to be used on a single thread. The same thread for
which you are trying to create a thread-state already has such a state.

Earlier versions of Python made no attempt to detect this, but as far as
everyone can tell, was always an implicit assumption.

Unfortunately, much of this stuff has never been thought through
correctly. Trying to work with multiple InterpreterState objects is
also very difficult, and in some cases simply does not work, as not all
global variables are stored in an InterpreterState. Theoretically
though, this is probably what you want - if a different InterpreterState
is current, I would expect a ThreadState specific to the
InterpreterState could be used - but I doubt it works <wink>

The easy answer is to stop trying to create multiple interpreter states,
then use the PyGILState calls to manage your thread-state. These should
be 2 lines per function (rather than the many that exist now). If you
need to build in both pre 2.3 and 2.3+:
#if (PY_VERSION_HEX >= 0x02030000)
#define USE_GILSTATE
#endif
/* Then */
#ifdef USE_GILSTATE
PyGILState_State state = PyGILState_Ensure();
#else
20 lines of existing code
#endif
// your body, and at the end
#ifdef USE_GILSTATE
PyGILState_Release(state);
#else
existing code
#endif

Mark.

 

[Mark Hammond]

Interesting. Unfortunately, there isn't a whole lot of information out
there on what we're trying to do.

I went ahead and commented out that check, but then I came across another
problem. In object.c, it hit an assertion:

assert(mro != NULL);

This was for a custom Python type I had created in C. It was assuming this
mro field was initialized, which it wasn't. It must be something new,
because this all used to work in 2.2

I will have to keep looking at the release notes.

I wish there was a "upgrading from 2.2 to 2.3" document for people who are
writing custom Python extensions, and doing embedding and such.

This assertion appears in PyObject_GenericGetAttr(). What was the
attribute being requested? I'm guessing your extension type is doing
something funky, as otherwise PyObject_GenericGetAttr() would not be called.

Almost all of my types upgraded without any pain. A few of the win32com
ones did require work to work with tp_iter, but that is only as these
types are themselves doing funky things.

I *think* that having tp_flags as zero works OK, but as soon as you use
Py_TPFLAGS_DEFAULT, you are expected to initialize your type
differently. Sorry, but I can recall any more details.

Mark.

 

[Paul Miller]

Unfortunately, much of this stuff has never been thought through

correctly. Trying to work with multiple InterpreterState objects is
also very difficult, and in some cases simply does not work, as not all
global variables are stored in an InterpreterState. Theoretically
though, this is probably what you want - if a different InterpreterState
is current, I would expect a ThreadState specific to the
InterpreterState could be used - but I doubt it works <wink>

It used to work perfectly in 2.2. Thus my dilemma.

The easy answer is to stop trying to create multiple interpreter states,
then use the PyGILState calls to manage your thread-state. These should
be 2 lines per function (rather than the many that exist now). If you
need to build in both pre 2.3 and 2.3+:

But, I'm not looking for a replacement for per-thread state. I am actually
maintaining multiple *interpreters*, each with all those global variables
you talked about. I need to do this because I want to be able to create an
"environment", load some scripts, run them, then delete that "environment"
without affecting other "environments".

If this PyGILState actually is an entire Python interpreter environment,
then I'll happily switch to it.

 

[Paul Miller]

assert(mro != NULL);

This assertion appears in PyObject_GenericGetAttr(). What was the
attribute being requested? I'm guessing your extension type is doing
something funky, as otherwise PyObject_GenericGetAttr() would not be called.

I am using GenericGetAttr() so my tp_getset list will work. This used to
work in 2.2.

I *think* that having tp_flags as zero works OK, but as soon as you use
Py_TPFLAGS_DEFAULT, you are expected to initialize your type
differently. Sorry, but I can recall any more details.

Nope, that didn't help. Since I am using new-style types, I had to have
most of those flags set anyway.

It seems to me that GenericGetAttr() should not assume mro to be
initialized. I haven't come across any sample custom types that initialize
it, and for whatever reason, it is not getting set up automatically.

 

[Paul Miller]

It seems to me that GenericGetAttr() should not assume mro to be

initialized. I haven't come across any sample custom types that initialize
it, and for whatever reason, it is not getting set up automatically.

I traced through my initialization code, where I am "readying" my custom
types, and it is failing in the mro_implementation() function in
typeobject.c. For whatever reason, the PySequence_List() call in this
function is failing.

I have no clue why at this point.