os.pipe and subprocess under Windows

[Jeremy Sanders]

Hi - I have some code which works under linux. It starts a remote python
process using subprocess and communicates to it via a pipe created by
os.pipe. As far as I understand, child processes should inherit file
descriptors from the parent if close_fds=False on the suprocess.Popen
command line.

This code doesn't work under Window, but gives "bad file descriptor" when
trying to read from the pipe in the child process. I have some example code
which fails. It consists of two files, master.py and slave.py. The file
descriptor for reading from the pipe is passed as a argument to slave.py.

----------------------------------------------------------------
#!/usr/bin/env python
# This is master.py

import os
import os.path
import sys
import subprocess

def runMaster():
# create pipe to communicate with remote process
rpipe, wpipe = os.pipe()

# start remote process
cmdline = [sys.executable,
os.path.join( os.path.dirname(
os.path.abspath(__file__)), 'slave.py' ),
str(rpipe) ]

remote = subprocess.Popen(cmdline, shell=False, bufsize=0,
close_fds=False)

# send text to remote process via pipe
os.write(wpipe, 'hi there$')

# wait until remote exit
remote.wait()

if __name__ == '__main__':
runMaster()

------------------------------------------------------------------

# This is slave.py

import sys
import os

def runSlave(fd):
"""Copy text to stderr from file descriptor until a $ symbol."""
while True:
intext = os.read(fd, 1)
if intext == '$':
break
elif intext:
# write text from pipe to stderr
sys.stderr.write('* %s\n' % intext)

if __name__ == '__main__':
fd = int(sys.argv[1])
runSlave(fd)

-------------------------------------------------------------------

Does anyone have any ideas how to get this to work under Windows? Is it
correct code under unix?

Thanks

Jeremy

 

[Lawrence D'Oliveiro]

As far as I understand, child processes should inherit file
descriptors from the parent if close_fds=False on the suprocess.Popen
command line.

This code doesn't work under Window, but gives "bad file descriptor" when
trying to read from the pipe in the child process.

<http://docs.python.org/library/subprocess.html>:

If close_fds is true, all file descriptors except 0, 1 and 2 will be
closed before the child process is executed. (Unix only). Or, on
Windows, if close_fds is true then no handles will be inherited by the
child process.

Windows has no fork(2).

 

[Jeremy Sanders]

<http://docs.python.org/library/subprocess.html>:

If close_fds is true, all file descriptors except 0, 1 and 2 will be
closed before the child process is executed. (Unix only). Or, on
Windows, if close_fds is true then no handles will be inherited by the
child process.

Windows has no fork(2).

Yes - I saw that - thanks. This suggests that as I have used
closed_fds=False, then the child process will inherit the handles under
Windows, which it doesn't seem to. This documentation looks wrong to me.

I know Windows has no fork - that's why I used subprocess. This MSDN page
suggests you you need to do something with SetHandleInformation to get them
to be inherited. Doesn't subprocess do that?

http://msdn.microsoft.com/en-us/library/ms682499(VS.85).aspx

 

[Jeremy Sanders]

Hi - I have some code which works under linux. It starts a remote python
process using subprocess and communicates to it via a pipe created by
os.pipe. As far as I understand, child processes should inherit file
descriptors from the parent if close_fds=False on the suprocess.Popen
command line.

Hmm... examining the code for os.pipe in posixmodule.c, it looks like pipes
are create specifically to be non-inheritable in Windows. I can't see why
you would want a non-inheritable pipe, so I would call this a bug.

I suppose I could try this trick from subprocess.py to make the pipes
inheritable:

def _make_inheritable(self, handle):
"""Return a duplicate of handle, which is inheritable"""
return DuplicateHandle(GetCurrentProcess(), handle,
GetCurrentProcess(), 0, 1,
DUPLICATE_SAME_ACCESS)

Pretty nasty to have to do this though, and I would have to add a win32api
dependency, or hack around with the _subprocess module.

Jeremy