J
Jonathan Harden
Hi,
We have a class which executes external processes in a controlled environment and does "things" specified by the client program with each line of output. To do this we have been attaching stdout from the subprocess.Popen to apseudo terminal (pty) made with pty.openempty and opened with os.fdopen. Inoticed that we kept getting a bunch of extra new line characters.
This is all using python 2.6.4 in a centos6 environment.
After some investigation I realised we needed to use universal_newline support so I enabled it for the Popen and specified the mode in the fdopen to be rU. Things still seemed to be coming out wrong so I wrote up a test program boiling it down to the simplest cases (which is at the end of this message). The output I was testing was this:
Fake\r\nData\r\n
as seen through hexdump -C:
0000000c
Now if I do a simple subprocess.Popen and set the stdout to subprocess.PIPE, then do p.stdout.read() I get the correct output of
Fake\nData\n
When do the Popen attached to a pty I end up with
Fake\n\nData\n\n
Does anyone know why the newline conversion would be incorrect, and what I could do to fix it? In fact if anyone even has any pointers to where this might be going wrong I'd be very helpful, I've done a lot of hours of fiddling with this and googling to no avail.
Thanks,
Jonathan
#!/usr/bin/env python2.6.4
import os
import pty
import subprocess
import select
import fcntl
class TestRead(object):
def __init__(self):
super(TestRead, self).__init__()
self.outputPipe()
self.outputPty()
def outputPipe(self):
p1 = subprocess.Popen(
("/bin/cat", "output.txt"),
stdout=subprocess.PIPE,
universal_newlines=True
)
print "1: %r" % p1.stdout.read()
def outputPty(self):
outMaster, outSlave = pty.openpty()
fcntl.fcntl(outMaster, fcntl.F_SETFL, os.O_NONBLOCK)
p2 = subprocess.Popen(
("/bin/cat", "output.txt"),
stdout=outSlave,
universal_newlines=True
)
with os.fdopen(outMaster, 'rU') as pty_stdout:
while True:
try:
rfds, _, _ = select.select([pty_stdout], [], [], 0.1)
break
except select.error:
continue
for fd in rfds:
buf = pty_stdout.read()
print "2: %r" % buf
if __name__ == "__main__":
t = TestRead()
We have a class which executes external processes in a controlled environment and does "things" specified by the client program with each line of output. To do this we have been attaching stdout from the subprocess.Popen to apseudo terminal (pty) made with pty.openempty and opened with os.fdopen. Inoticed that we kept getting a bunch of extra new line characters.
This is all using python 2.6.4 in a centos6 environment.
After some investigation I realised we needed to use universal_newline support so I enabled it for the Popen and specified the mode in the fdopen to be rU. Things still seemed to be coming out wrong so I wrote up a test program boiling it down to the simplest cases (which is at the end of this message). The output I was testing was this:
Fake\r\nData\r\n
as seen through hexdump -C:
00000000 46 61 6b 65 0d 0a 44 61 74 61 0d 0a |Fake..Data..|
0000000c
Now if I do a simple subprocess.Popen and set the stdout to subprocess.PIPE, then do p.stdout.read() I get the correct output of
Fake\nData\n
When do the Popen attached to a pty I end up with
Fake\n\nData\n\n
Does anyone know why the newline conversion would be incorrect, and what I could do to fix it? In fact if anyone even has any pointers to where this might be going wrong I'd be very helpful, I've done a lot of hours of fiddling with this and googling to no avail.
Thanks,
Jonathan
#!/usr/bin/env python2.6.4
import os
import pty
import subprocess
import select
import fcntl
class TestRead(object):
def __init__(self):
super(TestRead, self).__init__()
self.outputPipe()
self.outputPty()
def outputPipe(self):
p1 = subprocess.Popen(
("/bin/cat", "output.txt"),
stdout=subprocess.PIPE,
universal_newlines=True
)
print "1: %r" % p1.stdout.read()
def outputPty(self):
outMaster, outSlave = pty.openpty()
fcntl.fcntl(outMaster, fcntl.F_SETFL, os.O_NONBLOCK)
p2 = subprocess.Popen(
("/bin/cat", "output.txt"),
stdout=outSlave,
universal_newlines=True
)
with os.fdopen(outMaster, 'rU') as pty_stdout:
while True:
try:
rfds, _, _ = select.select([pty_stdout], [], [], 0.1)
break
except select.error:
continue
for fd in rfds:
buf = pty_stdout.read()
print "2: %r" % buf
if __name__ == "__main__":
t = TestRead()