Python deadlocks when Popen in multithread.

[INADA Naoki]

On Ubuntu 10.10 amd64 and dual-core CPU, following code deadlocks.
What's wrong on this code?

#http://codepad.org/GkoXHbik

#!/usr/bin/env python

from subprocess import *
from threading import *

THREAD_COUNT=50

def worker():
p = Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
out = p.communicate("hoge")[0]
print "%s %s" % (current_thread().name, out)

threads = []
for i in range(THREAD_COUNT):
threads.append(Thread(target=worker, name=str(i)))
for t in threads: t.start()
for t in threads: t.join()

 

[Jed Smith]

def worker():
   p = Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
   out = p.communicate("hoge")[0]
   print "%s %s" % (current_thread().name, out)

If, instead of spawning workers you directly call worker(), does it succeed?

I suspect the threading is a red herring here.

 

[INADA Naoki]

def worker():
   p = Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
   out = p.communicate("hoge")[0]
   print "%s %s" % (current_thread().name, out)

If, instead of spawning workers you directly call worker(), does it succeed?
Yes.

I suspect the threading is a red herring here.

But thread & subprocess is a normal way to invoke outer commands in
parallel.

 

[INADA Naoki]

I've found the bug http://bugs.python.org/issue1731717.
It seems wating subprocess can be multithreaded but creating
subprocess cannot.

def worker():
   p = Popen(["cat"], stdin=PIPE, stdout=PIPE, stderr=STDOUT)
   out = p.communicate("hoge")[0]
   print "%s %s" % (current_thread().name, out)

If, instead of spawning workers you directly call worker(), does it succeed?
Yes.

I suspect the threading is a red herring here.

But thread & subprocess is a normal way to invoke outer commands in
parallel.