PyGilState_Ensure interrupts python critical sections

[billy.omahony]

Hi,

I have a native windows thread in a c python module which calls into
python code and adds an item to a data structure (a home-grown
circular buffer). At the same time my main python application is
removing items from this data structure.

Unlike native python containers adding and removing items from the
circular buffer is not atomic. So I need to put access to it in a
critical section. Using threading.Lock will work for native python
threads competing for access. But when access is from a windows thread
this will not work. That is because my call to PyGilState_ensure will
preempt my native python thread ***even when it is inside the critical
section***.

What is going on looks something like this (I think).


Py Python Windows Py threading.Lock
resource
Sched Thread Thread Code | |
| | | | |
|Go (GIL)# | | | |
| # | | | |
| # | | | |
| #...Doit.....|...........>| | |
| # | |. acquire...>| |
|<-PyGILState_Ensure--| | | |
| ... ... ... ...
|Stop #
|-------`|
|
|----Ensure rtns-----># PyObject_ | |
| : |CallMethod | | |
| : |.(Doit)...> |. acquire...>| DEADLOCK |
:
:
:.how does python thread tell
PyScheduler not to give away
Gil until we are done with
critical section??

So my question is how in python do I tell the scheduler not to prempt
the current python thread. Especially how to tell it not to give the
GIL to any calls to PyGILState_ensure until further notice (i.e. until
I am outside my critical section?? It sounds like a reasonable request
- surely this exists already but I can't find it.

One thing that may work (though the documentation does not
specifically say so) is using setcheckinterval() to set the check
interval to a very large value, acessing my shared structure and then
setting the check interval back to the previous value. Provided my
access to the shared structure takes less byte codes than what I set
checkinterval to I should be okay. However that would be very
dependant on the exact fine detail of how the check interval works and
may not be compatible with other Python releases

Maybe someone familiar with the python source code would know for
sure?

I am using Python 2.4.3 on windows XP.

Thanks for any help/suggestions offered!
BR,
Billy.

 

[Chris Mellon]

Hi,

I have a native windows thread in a c python module which calls into
python code and adds an item to a data structure (a home-grown
circular buffer). At the same time my main python application is
removing items from this data structure.

Unlike native python containers adding and removing items from the
circular buffer is not atomic. So I need to put access to it in a
critical section. Using threading.Lock will work for native python
threads competing for access. But when access is from a windows thread
this will not work. That is because my call to PyGilState_ensure will
preempt my native python thread ***even when it is inside the critical
section***.

What is going on looks something like this (I think).


Py Python Windows Py threading.Lock
resource
Sched Thread Thread Code | |
| | | | |
|Go (GIL)# | | | |
| # | | | |
| # | | | |
| #...Doit.....|...........>| | |
| # | |. acquire...>| |
|<-PyGILState_Ensure--| | | |
| ... ... ... ...
|Stop #
|-------`|
|
|----Ensure rtns-----># PyObject_ | |
| : |CallMethod | | |
| : |.(Doit)...> |. acquire...>| DEADLOCK |
:
:
:.how does python thread tell
PyScheduler not to give away
Gil until we are done with
critical section??

So my question is how in python do I tell the scheduler not to prempt
the current python thread. Especially how to tell it not to give the
GIL to any calls to PyGILState_ensure until further notice (i.e. until
I am outside my critical section?? It sounds like a reasonable request
- surely this exists already but I can't find it.

It took me some time to understand the problem you were describing
because you've got some terms backward - there's no Python scheduler,
and calling PyGILState_ensure doesn't preempt anything. The Python
interpreter *releases* the GIL every so often to allow other threads
looking for it to run, but calling the python GIL functions has no
effect on preemption.

The problem is that the GIL is being released while your object lock
is held, a second thread (started from C) acquires the GIL and then
blocks on the object lock. What you seem to be seeing is that it
blocking on the object lock is preventing it from releasing the GIL,
which prevents the python thread from running and releasing the lock.

This shouldn't happen - blocking on a lock releases the GIL, so the
python thread should run, release the GIL, and eventually your C
thread should be able to acquire both locks at the same time. Are you
sure that you're correctly acquiring the GIL in your C code?


The data flow you *should* be seeing should look something like this:

GIL object lock (P=Python, C=C, *=released)
-------------------
P P Python holds both locks
* P Python releases the GIL, inside critical section
C P C thread acquires the GIL and starts executing Python code
* P C thread tries to acquire object lock and blocks,
releasing the GIL
P P Python thread re-acquires the GIL
P * Python thread exits critical section and releases
the object lock
* * Python thread releases the GIL (can be in any
order with next state)
* C The C thread acquires the object lock and blocks on the GIL
C C C thread acquires the GIL and continues execution.

 

[billy.omahony]

Many thanks for the clarification. Also good idea to focus on lock
ownership rather that thread activity in your diagram.

To be honest I was not actually experiencing deadlock issues. I had
just deduced (incorrectly) that I might do so if I started using locks
in my py code called from c-land. As I was suffering some horrible
race-condtions that had suddenly appeared in code that had been
perfectly stable for the past 6 months I didn't want to go down that
route given my misconceptions; Even if introducing the locks worked
I'd always feel it was a deadlock waiting to happen.

Misunderstanding arose as for some reason I had decided that python
code called from C did not allow any other python threads to run until
control had returned to C and the gil had been released from C-land.
I.e. once the gil was given to the C thread then python had to wait
until it was given back.

I have to say the docs could be a little more explicit on the
mechanisms involved. They tend to be better at stating what individual
functions do rather that giving overall explanations. Maybe I'll get
around to submitting a chapter on it

Hi,

I have a native windows thread in a c python module which calls into
python code and adds an item to a data structure (a home-grown
circular buffer). At the same time my main python application is
removing items from this data structure.

Unlike native python containers adding and removing items from the
circular buffer is not atomic. So I need to put access to it in a
critical section. Using threading.Lock will work for native python
threads competing for access. But when access is from a windows thread
this will not work. That is because my call to PyGilState_ensure will
preempt my native python thread ***even when it is inside the critical
section***.

What is going on looks something like this (I think).

Py Python Windows Py threading.Lock
resource
Sched Thread Thread Code | |
| | | | |
|Go (GIL)# | | | |
| # | | | |
| # | | | |
| #...Doit.....|...........>| | |
| # | |. acquire...>| |
|<-PyGILState_Ensure--| | | |
| ... ... ... ...
|Stop #
|-------`|
|
|----Ensure rtns-----># PyObject_ | |
| : |CallMethod | | |
| : |.(Doit)...> |. acquire...>| DEADLOCK |
:
:
:.how does python thread tell
PyScheduler not to give away
Gil until we are done with
critical section??

So my question is how in python do I tell the scheduler not to prempt
the current python thread. Especially how to tell it not to give the
GIL to any calls to PyGILState_ensure until further notice (i.e. until
I am outside my critical section?? It sounds like a reasonable request
- surely this exists already but I can't find it.

It took me some time to understand the problem you were describing
because you've got some terms backward - there's no Python scheduler,
and calling PyGILState_ensure doesn't preempt anything. The Python
interpreter *releases* the GIL every so often to allow other threads
looking for it to run, but calling the python GIL functions has no
effect on preemption.

The problem is that the GIL is being released while your object lock
is held, a second thread (started from C) acquires the GIL and then
blocks on the object lock. What you seem to be seeing is that it
blocking on the object lock is preventing it from releasing the GIL,
which prevents the python thread from running and releasing the lock.

This shouldn't happen - blocking on a lock releases the GIL, so the
python thread should run, release the GIL, and eventually your C
thread should be able to acquire both locks at the same time. Are you
sure that you're correctly acquiring the GIL in your C code?

The data flow you *should* be seeing should look something like this:

GIL object lock (P=Python, C=C, *=released)
-------------------
P P Python holds both locks
* P Python releases the GIL, inside critical section
C P C thread acquires the GIL and starts executing Python code
* P C thread tries to acquire object lock and blocks,
releasing the GIL
P P Python thread re-acquires the GIL
P * Python thread exits critical section and releases
the object lock
* * Python thread releases the GIL (can be in any
order with next state)
* C The C thread acquires the object lock and blocks on the GIL
C C C thread acquires the GIL and continues execution.

One thing that may work (though the documentation does not
specifically say so) is using setcheckinterval() to set the check
interval to a very large value, acessing my shared structure and then
setting the check interval back to the previous value. Provided my
access to the shared structure takes less byte codes than what I set
checkinterval to I should be okay. However that would be very
dependant on the exact fine detail of how the check interval works and
may not be compatible with other Python releases

Maybe someone familiar with the python source code would know for
sure?

I am using Python 2.4.3 on windows XP.

Thanks for any help/suggestions offered!
BR,
Billy.

 

[Chris Mellon]

Many thanks for the clarification. Also good idea to focus on lock
ownership rather that thread activity in your diagram.

To be honest I was not actually experiencing deadlock issues. I had
just deduced (incorrectly) that I might do so if I started using locks
in my py code called from c-land. As I was suffering some horrible
race-condtions that had suddenly appeared in code that had been
perfectly stable for the past 6 months I didn't want to go down that
route given my misconceptions; Even if introducing the locks worked
I'd always feel it was a deadlock waiting to happen.

A good reason to avoid the threading metaphor for concurrency entirely

Misunderstanding arose as for some reason I had decided that python
code called from C did not allow any other python threads to run until
control had returned to C and the gil had been released from C-land.
I.e. once the gil was given to the C thread then python had to wait
until it was given back.

This is essentially true. The part you're missing is that some Python
functions themselves release the GIL and call into C - in this case,
it's blocking on a lock. Some other functions that do this including
blocking on file or (especially) socket IO.

If this didn't happen, it'd be pretty much impossible to mix Python
threads and C threads in any safe manner - the deadlock situation you
envisioned would be a very common reality.