threading and iterator crashing interpreter

[Janto Dreijer]

I have been having problems with the Python 2.4 and 2.5 interpreters
on both Linux and Windows crashing on me. Unfortunately it's rather
complex code and difficult to pin down the source.

So I've been trying to reduce the code. In the process it's started to
crash in different ways. I'm not sure if any of it is related. The
following is only crashing Python 2.5 (r25:51908, Sep 19 2006,
09:52:17) [MSC v.1310 32 bit (Intel)] on win32 in two different(?)
ways.

====================

Using the login1() function will throw a bunch of exceptions the most
interesting of which is:

Exception in thread Thread-34:
Traceback (most recent call last):
File "C:\Python25\lib\threading.py", line 460, in __bootstrap
self.run()
File "C:\Python25\lib\threading.py", line 440, in run
self.__target(*self.__args, **self.__kwargs)
File "Copy of scratchpad.py", line 20, in login1
random.choice(System.sessions).session_iter.next()
RuntimeError: instance.__dict__ not accessible in restricted mode

I'm not concerned with the Python exceptions, but about 40% of the
time it will launch the Visual Studio JIT Debugger with an "Unhandled
exception at 0x1e03973e in pythonw.exe: 0xC0000005: Access violation
reading location 0x00002024."

=================

Using the login2() function will sometimes (about 30% of the time)
cause the following to be printed out:
"""
Unhandled exception in thread started by
Error in sys.excepthook:

Original exception was:
"""

===============

Here's the code. Will it be easier to find the problem if I tried
compiling Python myself?

from threading import Thread
import random

def session_iter(f):
# second interpreter
while 1:
yield len(f.__dict__)

class Session:
def __init__(self):
self.session_iter = session_iter(self)

class System:
sessions = []

def login1():
# first interpreter
System.sessions.append(Session())
while 1:
random.choice(System.sessions).session_iter.next()

def login2():
# first interpreter
System.sessions.append(Session())
System.sessions.pop().session_iter.next()

# main interpreter
threads = [Thread(target=login1) for t in range(2000)]
for thread in threads:
thread.start()

 

[Gabriel Genellina]

I have been having problems with the Python 2.4 and 2.5 interpreters
on both Linux and Windows crashing on me. Unfortunately it's rather
complex code and difficult to pin down the source.

So I've been trying to reduce the code. In the process it's started to
crash in different ways. I'm not sure if any of it is related. The
following is only crashing Python 2.5 (r25:51908, Sep 19 2006,
09:52:17) [MSC v.1310 32 bit (Intel)] on win32 in two different(?)
ways.

====================

Using the login1() function will throw a bunch of exceptions the most
interesting of which is:

Exception in thread Thread-34:
Traceback (most recent call last):
File "C:\Python25\lib\threading.py", line 460, in __bootstrap
self.run()
File "C:\Python25\lib\threading.py", line 440, in run
self.__target(*self.__args, **self.__kwargs)
File "Copy of scratchpad.py", line 20, in login1
random.choice(System.sessions).session_iter.next()
RuntimeError: instance.__dict__ not accessible in restricted mode

From the error message, you appear to be using some form of restricted
execution - RExec or similar. I didn't try that way, but using the normal
mode, I got this different exception instead (using login1):

Exception in thread Thread-85:
Traceback (most recent call last):
File "c:\apps\python\lib\threading.py", line 460, in __bootstrap
self.run()
File "c:\apps\python\lib\threading.py", line 440, in run
self.__target(*self.__args, **self.__kwargs)
File "crash.py", line 20, in login1
random.choice(System.sessions).session_iter.next()
ValueError: generator already executing

It appears to indicate that you must syncronize the generators.
Adding a Lock object to Session appears to work OK:

class Session:
def __init__(self):
self.lock = Lock()
self.session_iter = session_iter(self)

def login1():
# first interpreter
System.sessions.append(Session())
while 1:
session = random.choice(System.sessions)
session.lock.acquire()
session.session_iter.next()
session.lock.release()

Your login2 version does not generate any error on my PC.

 

[Bjoern Schliessmann]

I have been having problems with the Python 2.4 and 2.5
interpreters on both Linux and Windows crashing on me.

I don't understand -- does the interpreter crash (segfault) or is
just your program terminating due to unhandled exceptions?

Regards,


Björn

 

[Janto Dreijer]

I don't understand -- does the interpreter crash (segfault) or is
just your program terminating due to unhandled exceptions?

Segfault.

 

[Janto Dreijer]

En Sun, 11 Mar 2007 07:32:04 -0300, Janto Dreijer <[email protected]>
escribió:

I have been having problems with the Python 2.4 and 2.5 interpreters
on both Linux and Windows crashing on me. Unfortunately it's rather
complex code and difficult to pin down the source.

So I've been trying to reduce the code. In the process it's started to
crash in different ways. I'm not sure if any of it is related. The
following is only crashing Python 2.5 (r25:51908, Sep 19 2006,
09:52:17) [MSC v.1310 32 bit (Intel)] on win32 in two different(?)
ways.

====================

Using the login1() function will throw a bunch of exceptions the most
interesting of which is:

Exception in thread Thread-34:
Traceback (most recent call last):
File "C:\Python25\lib\threading.py", line 460, in __bootstrap
self.run()
File "C:\Python25\lib\threading.py", line 440, in run
self.__target(*self.__args, **self.__kwargs)
File "Copy of scratchpad.py", line 20, in login1
random.choice(System.sessions).session_iter.next()
RuntimeError: instance.__dict__ not accessible in restricted mode

From the error message, you appear to be using some form of restricted
execution - RExec or similar. I didn't try that way, but using the normal
mode, I got this different exception instead (using login1):

Exception in thread Thread-85:
Traceback (most recent call last):
File "c:\apps\python\lib\threading.py", line 460, in __bootstrap
self.run()
File "c:\apps\python\lib\threading.py", line 440, in run
self.__target(*self.__args, **self.__kwargs)
File "crash.py", line 20, in login1
random.choice(System.sessions).session_iter.next()
ValueError: generator already executing

It appears to indicate that you must syncronize the generators.
Adding a Lock object to Session appears to work OK:

class Session:
def __init__(self):
self.lock = Lock()
self.session_iter = session_iter(self)

def login1():
# first interpreter
System.sessions.append(Session())
while 1:
session = random.choice(System.sessions)
session.lock.acquire()
session.session_iter.next()
session.lock.release()

Your login2 version does not generate any error on my PC.

As far as I can tell I'm not running it from restricted mode
explicitly.

The reason I'm doing the random.choice() is so I don't have a handle
("session" in this case) that would prevent it from being garbage
collected. I am not bothered by the Python-level Exceptions. I am
bothered by the interpreter throwing a segfault. What I suspect is
happening is the Session object is deallocated, while it still has a
reference from within the iterator.

It's quite difficult to reproduce these bugs consistently. You might
need to run it a few times.

 

[Gabriel Genellina]

As far as I can tell I'm not running it from restricted mode
explicitly.

This error is rather strange then:

"restricted mode" means that the current builtins are not the standard
builtins.

The reason I'm doing the random.choice() is so I don't have a handle
("session" in this case) that would prevent it from being garbage
collected. I am not bothered by the Python-level Exceptions. I am
bothered by the interpreter throwing a segfault. What I suspect is
happening is the Session object is deallocated, while it still has a
reference from within the iterator.

But on your code Session objects are never deleted; they stay inside
System.sessions. Having an extra reference (the session variable) doesnt
matter.

It's quite difficult to reproduce these bugs consistently. You might
need to run it a few times.

At least, the problem of using the same generator from different threads
still remains, if you don't use my modified code. In general, when using
multiple threads you always need some way to syncronize access to shared
objects. You are lucky with Sessions because append is an atomic operation
on lists; for more information see
http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-thread-safe.htm

 

[Janto Dreijer]

This error is rather strange then:


"restricted mode" means that the current builtins are not the standard
builtins.

Googling says "Some googling suggests that this error is a hint that a
Python object created in one subinterpreter is being used in a
different subinterpreter."

But on your code Session objects are never deleted; they stay inside
System.sessions. Having an extra reference (the session variable) doesnt
matter.

Hmmm. You're right.

At least, the problem of using the same generator from different threads
still remains, if you don't use my modified code. In general, when using
multiple threads you always need some way to syncronize access to shared
objects. You are lucky with Sessions because append is an atomic operation
on lists; for more information see http://effbot.org/pyfaq/what-kinds-of-global-value-mutation-are-threa...

Again you're right. But even putting a try-except:return around the
random.choice still manages to break the interpreter. I'm not looking
for any meaningful output really. I just want it not to break.

Like I said, this is just watered down code to try and isolate the
problem.

 

[Gabriel Genellina]

Again you're right. But even putting a try-except:return around the
random.choice still manages to break the interpreter. I'm not looking
for any meaningful output really. I just want it not to break.

A try/except won't help a concurrency problem, unfortunately.

Like I said, this is just watered down code to try and isolate the
problem.

So I suggest to start with some small, *correct*, code, and then add more
features until you reconstruct your big program in a safe way.

 

[Rhamphoryncus]

Again you're right. But even putting a try-except:return around the
random.choice still manages to break the interpreter. I'm not looking
for any meaningful output really. I just want it not to break.

I think you need to clarify what you mean by "break". Python tries to
prevent C-level corruption (segfaults, etc), but does NOT attempt to
prevent python-level corruption. The python-level corruption will
*probably* be localized (and the objects involved could be deleted, no
harm done), but there's no guarantee that they won't involve some
global too.

But the examples you give involve C-level corruption. They're not
using some low-level interface either, so they're genuine bugs.

login1 did segfault for me, but the output is random and usually very
long. login2 is somewhat random as well, but much shorter and to the
point:

Exception exceptions.SystemError: '../Python/traceback.c:96: bad
argument to internal function' in <generator object at 0xa7956aec>
ignored
Exception exceptions.SystemError: '../Python/traceback.c:96: bad
argument to internal function' in <generator object at 0xa79569ac>
ignored
Exception exceptions.SystemError: '../Python/traceback.c:96: bad
argument to internal function' in <generator object at 0xa795656c>
ignored
Exception exceptions.SystemError: '../Python/traceback.c:96: bad
argument to internal function' in <generator object at 0xa795ebec>
ignored
Segmentation fault

Blah! I was trying to trace it down, but then the phase of the moon
changed and login2 (as well as my variations therein) stopped
failing. login1 still segfaults though. And login1 does sometimes
give me that weird restricted mode error, so you're not alone.

I do remember reading something about generators having a pointer to
the thread they're created in, and having problems when migrated
between threads. I don't have time to find the posts or bug reports
right now, but at least that gives you something to look for (before
filing a bug report).

Actually, I have a few more minutes.
http://mail.python.org/pipermail/python-dev/2006-October/069470.html
http://sourceforge.net/tracker/index.php?func=detail&aid=1579370&group_id=5470&atid=105470

That refers to a generator crash. You are using generators, but also
getting a weird dict error. Maybe related, maybe not.

I'll figure out if I've got a "fixed" version or not when I get back.

 

[Klaas]

http://sourceforge.net/tracker/index.php?func=detail&aid=1579370&grou...

That refers to a generator crash. You are using generators, but also
getting a weird dict error. Maybe related, maybe not.

I'll figure out if I've got a "fixed" version or not when I get back.

I was the one who filed the first bug. login2 is definitely the same
bug: you have a generator running in a thread, but the thread is being
garbage collected before the generator's .close() method is run (which
references the thread state).

Try the patch I posted in the bug above; if it works, then python
trunk/5.1maint should work.

-Mike

 

[Rhamphoryncus]

The bug report mentions 2.5.1, which hasn't be released yet, so
obviously I'm not using it.

I was the one who filed the first bug. login2 is definitely the same
bug: you have a generator running in a thread, but the thread is being
garbage collected before the generator's .close() method is run (which
references the thread state).

Try the patch I posted in the bug above; if it works, then python
trunk/5.1maint should work.

And with a fresh checkout of trunk I still get a crash with login1.
login2 isn't crashing, although it's not as consistent for me anyway.