How can I read streaming output of a subprocess

[Damjan Georgievski]

I want to read the stream of an external process that I start with
Python. From what I've seen that's not possible with the subprocess module?

I want to read the output of "ip monitor neigh" which will show changes
in the ARP table on my Linux computer. Each change is a new line printed
by "ip" and the process continues to run forever.

 

[Adam Skutt]

Try with
    cmd = 'your command here'
    stdout = Popen(cmd, shell = True, stdout = PIPE, stderr =STDOUT).stdout

One should never, ever start a process with shell=True unless it's
absolutely necessary. It is a waste of resources and an a security
issue waiting to happen. Just say no. Also, stderr shouldn't be
combined with stdout unless there is a need to mix the two streams,
which usually is not the case.

Also, one needs to hold on to the POpen object so the process can be
properly reaped later or use a with block. This is where things get
tricky for most applications. It gets a little extra tricky for 'ip
monitor'.

Assuming the intention is to write a command-line application, the
right thing to do _most likely_ is:
with Popen(['ip', 'monitor', 'neigh'], stdout=PIPE) as proc:
try:
for line in proc.stdout:
# Do your processing here, it need not be line-by-line.
finally:
process.terminate()

Calling process.terminate() explicitly is necessary since "ip monitor"
writes output forever. This is not the right thing to do in all
situations, and the right thing to do depends on your application and
the command being executed as a subprocess. It's not really possible
to generalize fully.

As such, your error-handling / cleanup code may end up being
completely different. Don't use my code if it's not applicable to
your situation. In any case, you need to cleanup after yourself,
which means waiting until the subprocess terminates. It may also
include cleaning up the file objects for the pipes. Using POpen
objects in a with block does both, so do that where you can.

The official Python documentation for the subprocess module is pretty
clear and worth reading. I encourage you to look through it
carefully. It does, unfortunately, gloss over which resources need to
be cleaned up and when.

Adam

 

[Damjan Georgievski]

I want to read the stream of an external process that I start with Python.

From what I've seen that's not possible with the subprocess module?

I want to read the output of "ip monitor neigh" which will show changes in
the ARP table on my Linux computer. Each change is a new line printed by
"ip" and the process continues to run forever.

You can use subprocess for this. If you don't call wait() or
communicate() on the Popen object, the process will happily continue
running. You can call readline on the stdout pipe, which will block
until the command you are running outputs (and flushes) a line.

Something like this should work:

p = subprocess.Popen(['ip', 'monitor', 'neigh'], stdout=subprocess.PIPE)
first_line = p.stdout.readline()

Buffering done by "ip" can ruin your day. It seems reasonable to
expect, though, that something that is printing state changes on
stdout will be careful to flush its output when appropriate.

thanks,

With your confirmation, I've rechecked again and now I see that the
problem I experienced before was with some strange ssh interaction.