Race condition deadlock in communicate when threading?

[Atherun]

In python 2.6.4 I have a fairly complex system running (copying and
pasting it would be quite difficult). At its core there are builders
that inherit from threading.Thread. I have some builders that run
external tasks via popen and read output using communicate. I have
the ability to run any number of builders in parallel. All of this
uses logging as well, where each builder has their own logger they
create from the base logger i.e. logging.getLogger("Logger.
%s"%self.Name).

When running several threads in parallel that each call popen and
communicate functions, python just hangs, about 95% of the time. All
of it, parent threads, any threads I started before I got to this
step, it just stops responding. Stepping through it with pdb trying
to find the deadlock will always make it work, killing the external
app I called via popen, after it has hung, will make it move along as
well. Looking at the stack traces using the code found
http://code.activestate.com/recipes/577334-how-to-debug-deadlocked-multi-threaded-programs/
the threads stop running with the following stacks, this is the last
output from the tracer before it stops responding.:

File: "c:\src\extern\python\lib\threading.py", line 497, in
__bootstrap
self.__bootstrap_inner()
File: "c:\src\extern\python\lib\threading.py", line 525, in
__bootstrap_inner
self.run()
File: "c:\src\extern\python\lib\threading.py", line 477, in run
self.__target(*self.__args, **self.__kwargs)
File: "c:\src\extern\python\lib\subprocess.py", line 877, in
_readerthread
buffer.append(fh.read())

And

out, err = proc.communicate("change: new\ndescription: %s
\n"%changelistDesc)
File: "c:\src\extern\python\lib\subprocess.py", line 689, in
communicate
return self._communicate(input)
File: "c:\src\extern\python\lib\subprocess.py", line 903, in
_communicate
stdout_thread.join()
File: "c:\src\extern\python\lib\threading.py", line 637, in join
self.__block.wait()
File: "c:\src\extern\python\lib\threading.py", line 237, in wait
waiter.acquire()

I'm trying to track this down so I can eliminate it for good as it
pops up in multiple places from time to time.

Any tips would be appreciated.

 

[Carl Banks]

There's really not enough information for us to debug this, but one possibility is that your subprocess is using buffered I/O; you're expecting the external task to write a string, but it doesn't actually write that string because it's sitting in a buffer. First thing to try is to see if the program accepts some kind of command line argument to run in unbuffered mode (forinstance, if you are calling a Python interpreter you can pass it the -u switch to force unbuffered I/O). If (like most programs) it doesn't have anoption to disable buffering, you can try running it on a pty device (if you're on Unix). If worse comes to worst, see if there's a way to get the external task to print lots of extra output (a verbosity setting, for instance); that could work in a pinch until you can debug it more thoroughly.


Carl Banks

 

[Wong Wah Meng-R32813]

Hello all,

I encounter this issue whereby I am not able to load cPickle module into the python I newly built. There is no issue when I load it right from the folder where the python executable and libpython2.7.so is built. However, it gives me this error when I load the same module using the installed files (python and all its modules, shared library from default /use/local folder that contains bin, lib, include sub-folders) from "make install" command.

Does anyone know why? Here is the error:-

$ python
Python 2.7.1 (r271:86832, Sep 27 2011, 15:19:26) [C] on hp-ux11
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '_extension_registry'

 

[Wong Wah Meng-R32813]

I just built python 2.7.1 on my HP Itanium 64-bit platform, using aCC. I encountered following issue when importing the random module. Does anyone know why am I getting this error? Thanks in advance for your reply.

$ python
Python 2.7.1 (r271:86832, Sep 27 2011, 15:19:26) [C] on hp-ux11
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "random.py", line 49, in <module>
import hashlib as _hashlib
File "hashlib.py", line 136, in <module>
globals()[__func_name] = __get_hash(__func_name)
File "hashlib.py", line 74, in __get_builtin_constructor
import _sha256
ImportError: No module named _sha256

 

[Chris Rebert]

I just built python 2.7.1 on my HP Itanium 64-bit platform, using aCC. I encountered following issue when importing the random module. Does anyone know why am I getting this error? Thanks in advance for your reply.

 File "hashlib.py", line 74, in __get_builtin_constructor
   import _sha256
ImportError: No module named _sha256

Did you get a message along the lines of "Python build finished, but
the necessary bits to build these modules were not found:" when you
built Python?

Cheers,
Chris

 

[Wong Wah Meng-R32813]

Yes I did. In fact earlier "make test" failed to execute due to _sha256 notfound in one of the module testing.

Earlier I ignored these failures from the build and make test as I thought they are not needed by my application. Now I can't use "random" module in my application. So I need to solve this issue.

Here is the output of the build.

Python build finished, but the necessary bits to build these modules were not found:
_bsddb _curses _curses_panel
_sqlite3 _ssl bsddb185
bz2 dl gdbm
imageop linuxaudiodev ossaudiodev
readline spwd sunaudiodev
zlib
To find the necessary bits, look in setup.py in detect_modules() for the module's name.


Failed to build these modules:
_ctypes termios

Regards,
Wah Meng

-----Original Message-----
From: (e-mail address removed) [mailto:[email protected]] On Behalf Of Chris Rebert
Sent: Tuesday, September 27, 2011 5:52 PM
To: Wong Wah Meng-R32813
Cc: (e-mail address removed)
Subject: Re: Error 'No module named _sha256' when importing random in python 2.7.1

I just built python 2.7.1 on my HP Itanium 64-bit platform, using aCC. I encountered following issue when importing the random module. Does anyone know why am I getting this error? Thanks in advance for your reply.

 File "hashlib.py", line 74, in __get_builtin_constructor
   import _sha256
ImportError: No module named _sha256

Did you get a message along the lines of "Python build finished, but
the necessary bits to build these modules were not found:" when you
built Python?

Cheers,
Chris

 

[Gabriel Genellina]

En Tue, 27 Sep 2011 06:08:54 -0300, Wong Wah Meng-R32813

Hello all,

I encounter this issue whereby I am not able to load cPickle module into
the python I newly built. There is no issue when I load it right from
the folder where the python executable and libpython2.7.so is built.
However, it gives me this error when I load the same module using the
installed files (python and all its modules, shared library from default
/use/local folder that contains bin, lib, include sub-folders) from
"make install" command.

Does anyone know why? Here is the error:-

$ python
Python 2.7.1 (r271:86832, Sep 27 2011, 15:19:26) [C] on hp-ux11
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '_extension_registry'

Looking at cPickle.c, it imports the copy_reg module and then looks for
its "_extension_registry" attribute. Maybe your copy_reg.py is broken, or
you have another copy_reg.py hiding the standard one.

 

[Wong Wah Meng-R32813]

Thanks Gabriel,

Indeed, there is corrupted copy_reg.py file residing in my $PYTHONPATH folder, as opposed to the standard one from the installed package. Thanks a lotfor your helps!

Regards,
Wah Meng

-----Original Message-----
From: [email protected] [mailto[email protected]] On Behalf Of Gabriel Genellina
Sent: Wednesday, September 28, 2011 7:53 AM
To: (e-mail address removed)
Subject: Re: Error 'module' object has no attribute "_extension_registry" when cPickle is imported from an installed Python 2.7.1

En Tue, 27 Sep 2011 06:08:54 -0300, Wong Wah Meng-R32813

Hello all,

I encounter this issue whereby I am not able to load cPickle module into
the python I newly built. There is no issue when I load it right from
the folder where the python executable and libpython2.7.so is built.
However, it gives me this error when I load the same module using the
installed files (python and all its modules, shared library from default
/use/local folder that contains bin, lib, include sub-folders) from
"make install" command.

Does anyone know why? Here is the error:-

$ python
Python 2.7.1 (r271:86832, Sep 27 2011, 15:19:26) [C] on hp-ux11
Type "help", "copyright", "credits" or "license" for more information.Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '_extension_registry'

Looking at cPickle.c, it imports the copy_reg module and then looks for
its "_extension_registry" attribute. Maybe your copy_reg.py is broken, or
you have another copy_reg.py hiding the standard one.

 

[Wong Wah Meng-R32813]

Hello there,

I couldn't detect this problem until I run my application that utilizes thread module in python that I just built on HP-UX 11.31 ia64 using aCC.

Could it be the build did not include enable thread option? _REENTRANT as stated in the README file? If yes, it looks like threading may not work "outof the box".

Traceback (most recent call last):

Excerpts from README file for python 2.7 build

HP-UX: When using threading, you may have to add -D_REENTRANT to the
OPT variable in the top-level Makefile; reported by Pat Knight,
this seems to make a difference (at least for HP-UX 10.20)
even though pyconfig.h defines it. This seems unnecessary when
using HP/UX 11 and later - threading seems to work "out of the
box".

 

[Wong Wah Meng-R32813]

Hello there,

I included --with-threads option into the configure script calling, which it failed to add the -D_REENTRANT into Makefile. So I manually added this into OPT of the Makefile. I ran make again and generated a new python binary.However, the new binary is still not able to start a new thread from the thread module.

I reviewed the config.log file, I think this failure has caused the build with threads failed. Cthreads.h header file is missing. Should this be included in my HP UX Compiler or it comes from python source? In python source Ionly found pythread.h. Anyway, I am only guessing what is wrong.

Hope to hear some feedback here, whether this is something missing or a bug..

configure:8407: checking for --with-threads
configure:8427: result: yes
configure:8484: checking for _POSIX_THREADS in unistd.h
configure:8503: result: yes
configure:8508: checking cthreads.h usability
configure:8508: cc +DD64 -I/home/r32813/local/include -c -g conftest.c >&5
"conftest.c", line 128: error #3696-D: cannot open source file "cthreads.h"
#include <cthreads.h>
^
1 error detected in the compilation of "conftest.c".

| #define HAVE_LIBDLD 1
| #define _REENTRANT 1
| /* end confdefs.h. */
| #include <cthreads.h>
configure:8508: result: no
configure:8508: checking for cthreads.h
configure:8508: result: no
configure:8521: checking mach/cthreads.h usability
configure:8521: cc +DD64 -I/home/r32813/local/include -c -g conftest.c >&5
"conftest.c", line 128: error #3696-D: cannot open source file "mach/cthreads.h"
#include <mach/cthreads.h>
^

1 error detected in the compilation of "conftest.c".
configure:8521: $? = 2
configure: failed program was:
| /* confdefs.h */
| #define _GNU_SOURCE 1

| #define HAVE_LIBDLD 1
| #define _REENTRANT 1
| /* end confdefs.h. */
| #include <mach/cthreads.h>
configure:8521: result: no
configure:8521: checking for mach/cthreads.h
configure:8521: result: no
configure:8533: checking for --with-pth
configure:8548: result: no
configure:8556: checking for pthread_create in -lpthread
configure:8572: cc +DD64 -I/home/r32813/local/include -o conftest -g -L/home/r32813/local/lib -L/home/r32813/Build/2.7.1/Python-2.7.1 conftest.c -l
nsl -lrt -ldld -ldl -lpthread >&5


Regards,
Wah Meng
Genesis Wafermap Support Ticket:
To report a problem: http://dyno.freescale.net/Question/QuestionMain3.asp?location=zmy02&category=&tickettype=6820
To request a service: http://dyno.freescale.net/Question/Questionmain3.asp?location=74&category=2&tickettype=6819
Or if it is related to EWM or DSA: http://dyno.freescale.net/Question/Questionmain3.asp?location=ZMY02&tickettype=6539

-----Original Message-----
From: Wong Wah Meng-R32813
Sent: Wednesday, September 28, 2011 4:37 PM
To: (e-mail address removed)
Subject: python 2.7.1 built not supporting thread?


Hello there,

I couldn't detect this problem until I run my application that utilizes thread module in python that I just built on HP-UX 11.31 ia64 using aCC.

Could it be the build did not include enable thread option? _REENTRANT as stated in the README file? If yes, it looks like threading may not work "outof the box".

Traceback (most recent call last):

Excerpts from README file for python 2.7 build

HP-UX: When using threading, you may have to add -D_REENTRANT to the
OPT variable in the top-level Makefile; reported by Pat Knight,
this seems to make a difference (at least for HP-UX 10.20)
even though pyconfig.h defines it. This seems unnecessary when
using HP/UX 11 and later - threading seems to work "out of the
box".