Capturing stderr and stdout of a subprocess as a single stream

[Fuzzyman]

Hello all,

Before I ask the question a couple of notes :

* This question is for implementing a script inside the Wing IDE. For
some reason using the subprocess module doesn't work so I need a
solution that doesn't use this module.
* The platform is Windows and I'm happy with a Windoze only solution.


I would like to execute subprocesses asynchronously and capture stdout
/ stderr as a single stream.

If I use "os.popen3(executable)" it gives me separate pipes for stdout
and stderr, but reads from them are blocking.

The output on stdout and stderr may be interleaved and I would like to
display them *as* they arrive. That means I can't just read from them
and output the results.

The only solution I can think of is to read from both a character at a
time on two separate threads, putting the data into a queue. A separate
thread could pull characters off the queue and display them. (I don't
need to differentiate between stdout and stderr when I display.)

Can anyone think of a better solution ?

My current code works, but *doesn't* capture stderr :

from threading import Thread

pipe = os.popen(executable)

def DisplayOutput():
while True:
output = pipe.read(1)
if not output:
break
display(output)

Thread(target=DisplayOutput).start()

All the best,


Fuzzyman
http://www.voidspace.org.uk/python/articles.shtml

 

[Gabriel Genellina]

Hello all,

Before I ask the question a couple of notes :

* This question is for implementing a script inside the Wing IDE. For
some reason using the subprocess module doesn't work so I need a
solution that doesn't use this module.
* The platform is Windows and I'm happy with a Windoze only solution.


I would like to execute subprocesses asynchronously and capture stdout
/ stderr as a single stream.

If I use "os.popen3(executable)" it gives me separate pipes for stdout
and stderr, but reads from them are blocking.

The output on stdout and stderr may be interleaved and I would like to
display them *as* they arrive. That means I can't just read from them
and output the results.

The only solution I can think of is to read from both a character at a
time on two separate threads, putting the data into a queue. A separate
thread could pull characters off the queue and display them. (I don't
need to differentiate between stdout and stderr when I display.)

Can anyone think of a better solution ?

My current code works, but *doesn't* capture stderr :

from threading import Thread

pipe = os.popen(executable)

def DisplayOutput():
while True:
output = pipe.read(1)
if not output:
break
display(output)

Thread(target=DisplayOutput).start()

All the best,

Fuzzymanhttp://www.voidspace.org.uk/python/articles.shtml

Try using popen4 instead. But since you already have a thread, it may
be better to use popen2 and two threads to read from stdout and stderr
to avoid a potential deadlock; they can put read lines into a Queue,
and DisplayOutput just get these lines in order. (See the warnings in
the popen2 module documentation).

 

[Fuzzyman]

Try using popen4 instead. But since you already have a thread, it may
be better to use popen2 and two threads to read from stdout and stderr
to avoid a potential deadlock; they can put read lines into a Queue,
and DisplayOutput just get these lines in order. (See the warnings in
the popen2 module documentation).

popen4 works great, I didn't even know it existed.

Two threads and a queue sounds horrible.

Thanks

Fuzzyman
http://www.voidspace.org.uk/python/articles.shtml

 

[Gabriel Genellina]

Two threads and a queue sounds horrible.

But unfortunately it's the only way if you don't control how the child
process behaves.
(It's not soooo ugly afterwards... certainly would be worse if you had
to syncronize both reading threads and the display thread using
semaphores by hand.)