How to catch a line with Popen

[TheSaint]

Hello.
I'm looking into subprocess.Popen docs.
I've launch the program with its arguments and that's smooth. I'm expecting
to read the output by *comunicate()* at every line that prgram may blow
during the process, but the output is given only when the child process is
ended.
I'd like to process the lines to display an information in percentage during
the running time of the child. Sorry but I'm poor of know-how, so I'm stuck
over to find a clue.

 

[Nobody]

I'm looking into subprocess.Popen docs. I've launch the program with its
arguments and that's smooth. I'm expecting to read the output by
*comunicate()* at every line that prgram may blow during the process, but
the output is given only when the child process is ended.

..communicate() reads until EOF, .wait()s for the process to terminate,
then returns the output as a pair of strings.

If you want to read data while the process is running, use the process'
..stdout and/or .stderr members, e.g.:

p = subprocess.Popen(...)
for line in p.stdout:
...
p.wait()

Don't forget to .wait() on the process when you're done, regardless of
whether you actually need the exit code.

 

[TheSaint]

Are you specifying a buffer size in the Popen command? If not, then the
Python side of things is unbuffered

The buffer is as per default. The program reports one line around 1/2 second
time.
I think I'll look into the option as Nobody states:

p = subprocess.Popen(...)
for line in p.stdout:
...
p.wait()

It is strange that would take a for cycle, rather than catching the line on-
the-fly. I can judge it now, I'm going to try it out.

 

[Chris Rebert]

The buffer is as per default. The program reports one line around 1/2 second
time.
I think I'll look into the option as Nobody states:

       p = subprocess.Popen(...)
       for line in p.stdout:
           ...
       p.wait()

It is strange that would take a for cycle, rather than catching the line on-
the-fly. I can judge it now, I'm going to try it out.

What do you mean by "on-the-fly" in this context?

Cheers,
Chris

 

[TheSaint]

What do you mean by "on-the-fly" in this context

I just suppose to elaborate the latest line, as soon it's written on the
pipe, and print some result on the screen.
Imaging something like

p= Popen(['ping','-c40','www.google.com'], stdout=PIPE)
for line in p.stdout:
print(str(line).split()[7])

I'd like to see something like *time=54.4*
This is just an example, where if we remove the "-c40" on the command line,
I'd expect to read the latest line(s), until the program will be killed.

 

[TheSaint]

I just suppose to elaborate the latest line, as soon it's written on the
pipe, and print some result on the screen.

I think some info is also here:
http://alexandredeverteuil.blogspot.com/

 

[Chris Torek]

Chris Rebert wrote:
I just suppose to elaborate the latest line, as soon it's written on the
pipe, and print some result on the screen.
Imaging something like

p= Popen(['ping','-c40','www.google.com'], stdout=PIPE)
for line in p.stdout:
print(str(line).split()[7])

I'd like to see something like *time=54.4*
This is just an example, where if we remove the "-c40" on the command line,
I'd expect to read the latest line(s), until the program will be killed.

In at least some versions of Python 2, file-like object "next"
iterators do not "work right" with unbuffered (or line-buffered)
pipe-file-objects. (This may or may not be fixed in Python 3.)

A simple workaround is a little generator using readline():

def line_at_a_time(fileobj):
"""
Return one line at a time from a file-like object.
Works around the iter behavior of pipe files in
Python 2.x, e.g., instead of "for line in file" you can
write "for line in line_at_a_time(file)".
"""
while True:
line = fileobj.readline()
if not line:
return
yield line

Adding this to your sample code gives something that works for me,
provided I fiddle with it to make sure that the only lines
examined are those with actual ping times:

p = subprocess.Popen(["ping", "-c5", "www.google.com"],
stdout = subprocess.PIPE)
for lineno, line in enumerate(line_at_a_time(p.stdout)):
if 1 <= lineno <= 5:
print line.split()[6]
else:
print line.rstrip('\n')
p.wait() # discard final result

(Presumably the enumerate() trick would not be needed in whatever
you really use.)

 

[TheSaint]

In at least some versions of Python 2

I'm with P3k . However thank you for your guidelines.
Last my attempt was to use a *for* p.wait() , as mentioned earlier

That looks good enough. I noted some little delay for the first lines,
mostly sure Popen assign some buffer even it is not set.

Haven't you try a perpetual ping, how would be the line_at_a_time ?

 

[Chris Torek]

[the "file"-type object iterators behave badly with pipes]

(This may still be true in Python 3, I just have no experience with
Py3k. "At least some version of Python 2" means "the ones I have
access to, and have tried." )

I'm with P3k . However thank you for your guidelines.
Last my attempt was to use a *for* p.wait() , as mentioned earlier

If you have a process that has not yet terminated and that you
must stop from your own python program, calling the wait() method
will wait forever (because you are now waiting for yourself, in
effect -- waiting for yourself to terminate the other process).

The only time to call p.wait() (or p.communicate(), which itself
calls the wait() method) is when you believe the subprocess is
on its wao to terminating -- in this case, after you force it
to do so.

That looks good enough. I noted some little delay for the first lines,
mostly sure Popen assign some buffer even it is not set.

According to the documentation, the default buffer size of Python 2
is 0, which is passed to fdopen() and makes the resulting files
unbuffered. I recall some sort of changes discussed for Py3k though.

Haven't you try a perpetual ping, how would be the line_at_a_time ?

Since it is a generator that only requests another line when called,
it should be fine. (Compare to the itertools "cycle" and "repeat"
generators, for instance, which "return" an infinite sequence.)

 

[TheSaint]

Since it is a generator that only requests another line when called,
it should be fine

Is it, then, that until the new itaration, the callee is on pause?