subprocess -popen - reading stdout from child - hangs

[gregpinero]

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()
while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:

But it just hangs at read()

proc.communicate() also just hangs. What am I doing wrong? Please
advise.

Thanks,

Greg

 

[gregpinero]

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()
while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:


But it just hangs at read()

proc.communicate() also just hangs. What am I doing wrong? Please
advise.

Well, using this subclass of subprocess.Popen fixes the problem
(http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554)

I don't understand how it works though. Would anyone mind
explaining? I'm thinking there's something fundamental about Unix
processes I'm not understanding.

-Greg

 

[Suresh Babu Kolla]

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()
while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:

From python documentation

`read([size])'
Read at most SIZE bytes from the file (less if the read hits `EOF'
before obtaining SIZE bytes). If the SIZE argument is negative or
omitted, read all data until `EOF' is reached. The bytes are
returned as a string object. An empty string is returned when
`EOF' is encountered immediately. (For certain files, like ttys,
it makes sense to continue reading after an `EOF' is hit.) Note
that this method may call the underlying C function `fread()' more
than once in an effort to acquire as close to SIZE bytes as
possible. Also note that when in non-blocking mode, less data than
what was requested may be returned, even if no SIZE parameter was
given.

read call in your code is waiting for EOF, since the script never exits
EOF is not reached.

Change read code to

proc.stdout.readline()

or

remove while 1 loop from loop.py.

HTH
Kolla

 

[Karthik Gurusamy]

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()

Add sys.stdout.close()

while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:


But it just hangs at read()

proc.communicate() also just hangs. What am I doing wrong? Please
advise.

Since your loop.py is still alive and hasn't closed its stdout, the
caller continues to wait for EOF (it doesn't know if loop.py is done
generating all its output)

Karthik

 

[Lawrence D'Oliveiro]

In message <[email protected]>,

Let's say I have this Python file called loop.py:

import sys
print 'hi'
sys.stdout.flush()
while 1:
pass

And I want to call it from another Python process and read the value
'hi'. How would I do it?

So far I have tried this:


But it just hangs at read()

That's because you didn't tell it how much to read, so by default it tries
to read until EOF <http://docs.python.org/lib/bltin-file-objects.html>. But
since the subprocess is still running and hasn't closed its stdout, the
pipe has not reached EOF.

 

[Lawrence D'Oliveiro]

In message <[email protected]>,

Well, using this subclass of subprocess.Popen fixes the problem
(http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554)

I don't understand how it works though. Would anyone mind
explaining? I'm thinking there's something fundamental about Unix
processes I'm not understanding.

The "select" calls are checking that the subprocess is ready to read before
writing to its stdin, or that it has something to write before reading from
its stdout.

The O_NONBLOCK fcntl simply says not to wait if there is nothing more to be
read, just return what was already read.

 

[gregpinero]

Add sys.stdout.close()

Adding sys.stdout.close() and removing sys.stdout.flush() seems to
make it work. But can the while loop still use sys.stdout later on?
Do I have to reopen it?

Thanks,

Greg

 

[Karthik Gurusamy]

Adding sys.stdout.close() and removing sys.stdout.flush() seems to
make it work. But can the while loop still use sys.stdout later on?
Do I have to reopen it?

Once you close a file-object, you cannot use it. You'll get exception
if you try.
I quickly tried the fdopen on id 1 (stdout in unix like OS) and it
seems to work
Traceback (most recent call last):

BUT you may want to re-think your design. Note that your caller
process will anyway stop reading further from your loop.py process
the moment it sees the "first" EOF. So even if you enable loop.py to
generate more output (thru' the above fdopen), the caller process is
not going to see the newly generated data.

If your requirement is to collect all output from loop.py then you
can't do it if loop.py has an infinite loop across its data generation
points (ie it can generate data after the infinite loop -- which
anyway doesn't sense).

If all you want is not to get blocked, try one of the select solutions
or read a small amount at a time (which you can guarantee to be
available). Yet another solution would be you could set up an alarm
and get out of the blocking read if the alarm fires.

Karthik