How can I know how much to read from a subprocess

[Guest]

Hello,

I want to write a terminal program in pygtk. It will run a subprocess,
display everything it writes in its standard output and standard
error, and let the user write text into its standard input.

The question is, how can I know if the process wrote something to its
output, and how much it wrote? I can't just call read(), since it will
block my process.

Thanks,
Noam

 

[Guest]

Ok, I could have researched this before posting, but here's an
explanation how to do it with twisted:

http://unpythonic.blogspot.com/2007/08/spawning-subprocess-with-pygtk-using.html

It seems that another solution is gobject.io_add_watch, but I don't
see how it tells me how much I can read from the file - if I don't
know that, I won't know the argument to give to the read() method in
order to get all the data:

http://www.pygtk.org/docs/pygobject/gobject-functions.html#function-gobject--io-add-watch

Noam

 

[A.T.Hofkamp]

It seems that another solution is gobject.io_add_watch, but I don't
see how it tells me how much I can read from the file - if I don't
know that, I won't know the argument to give to the read() method in
order to get all the data:

http://www.pygtk.org/docs/pygobject/gobject-functions.html#function-gobject--io-add-watch

Usually, gobject only tells you that data is there (that is all it knows).
Therefore a read(1) should be safe.
If that is too slow, consider os.read() which reads all data available (afaik,
never tried it myself).

Albert

 

[Guest]

Usually, gobject only tells you that data is there (that is all it knows).
Therefore a read(1) should be safe.

But even if it's fast enough, how do you know how many times you
should call read(1)? If you do it too much, you'll be blocked until
more output is available.

If that is too slow, consider os.read() which reads all data available (afaik,
never tried it myself).

I tried it now, and it blocks just like the normal file.read().

Thanks,
Noam

 

[Grant Edwards]

But even if it's fast enough, how do you know how many times you
should call read(1)? If you do it too much, you'll be blocked until
more output is available.

You don't know. That's why you use non-blocking mode.

 

[Karthik Gurusamy]

Hello,

I want to write a terminal program in pygtk. It will run a subprocess,
display everything it writes in its standard output and standard
error, and let the user write text into its standard input.

The question is, how can I know if the process wrote something to its
output, and how much it wrote? I can't just call read(), since it will
block my process.

The solution is not simple. You may try a separate thread to do the
reading (so that you can afford to block that single thread). You can
also run into other I/O deadlocks with a simple solution (ie if you
try to feed the whole stdin before reading any stdout, your stdin
feeding can block because the process's stdout is full -- it's a bit
confusing but for large stdin/stdout, a deadlock is very much
possible).

Did you try subprocess.Popen.communicate() ? Under the covers it
should take care of th ese deadlock issues. It may read and buffer the
whole stdout/stderr, so don't use it if the process generates infinite/
very-large stdout/stderr.

If the process you launch needs two way communication, like an ssh/ftp
session (where the stdin depends on process's prior stdout), the only
viable solution is to simulate a human (e.g. pexpect module)

Karthik

 

[Lawrence D'Oliveiro]

In message <[email protected]>,

... how can I know if the process wrote something to its
output, and how much it wrote?

Others have mentioned using one-byte reads. To tell if something is
available for reading, use select
<http://docs.python.org/lib/module-select.html>.

 

[A.T.Hofkamp]

But even if it's fast enough, how do you know how many times you
should call read(1)? If you do it too much, you'll be blocked until
more output is available.

after reading 1 byte, wait for gobject again.

In other words, when gobject sends a notice, read 1 byte.
When there is more, gobject will tell you (probably very often ).

I tried it now, and it blocks just like the normal file.read().

So "os.read(handle.fileno(), 1000000)" blocks after gobject tells you there is data?
That is different than I heard, but you are probably correct, I cannot easily test this.


Albert