non-blocking PIPE read on Windows

[placid]

Hi all,

I have been looking into non-blocking read (readline) operations on
PIPES on windows XP and there seems to be no way of doing this. Ive
read that you could use a Thread to read from the pipe, but if you
still use readline() wouldnt the Thread block too?

What i need to do is, create a process using subprocess.Popen, where
the subprocess outputs information on one line (but the info
continuesly changes and its always on the same line) and read this
information without blocking, so i can retrieve other data from the
line i read in then put this in a GUI interface.


readline() blocks until the newline character is read, but when i use
read(X) where X is a number of bytes then it doesnt block(expected
functionality) but i dont know how many bytes the line will be and its
not constant so i cant use this too.

Any ideas of solving this problem?


Cheers

 

[Simon Forman]

Hi all,

I have been looking into non-blocking read (readline) operations on
PIPES on windows XP and there seems to be no way of doing this. Ive
read that you could use a Thread to read from the pipe, but if you
still use readline() wouldnt the Thread block too?

Yes it will, but that's ok. In this case that's what it's for. While
the thread waits for the readline(), the rest of your program continues
to carry on.

 

[Dennis Lee Bieber]

readline() blocks until the newline character is read, but when i use
read(X) where X is a number of bytes then it doesnt block(expected
functionality) but i dont know how many bytes the line will be and its
not constant so i cant use this too.

Any ideas of solving this problem?

Use a thread that reads one character at a time; when it sees
whatever signals "end of line" (it sounds like you're reading a progress
bar implemented via <cr>overwrite). Combine the characters into a
string, return the string to the main program via a queue.

If there is no such "end of line" character, but there IS a
noticeable delay between "writes", a more complex method might suffice
-- in which one thread does the byte reads, setting a time value on each
read; a related thread then does a sleep() loop, checking the "last read
time" against the pause length -- if close enough to the pause duration,
combine and return...

Alternatively, take a good old style terminal keyboard (a VT100
Tempest-rated model should be ideal), and use it to beat Bill Gates over
the head until he agrees to push a high-priority upgrade to the command
line I/O system... or makes files work with select() (so you can combine
the time-out with the byte read)

--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[Antonio Valentino]

Hi all,

I have been looking into non-blocking read (readline) operations on
PIPES on windows XP and there seems to be no way of doing this. Ive
read that you could use a Thread to read from the pipe, but if you
still use readline() wouldnt the Thread block too?
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

What i need to do is, create a process using subprocess.Popen, where
the subprocess outputs information on one line (but the info
continuesly changes and its always on the same line) and read this
information without blocking, so i can retrieve other data from the
line i read in then put this in a GUI interface.


readline() blocks until the newline character is read, but when i use
read(X) where X is a number of bytes then it doesnt block(expected
functionality) but i dont know how many bytes the line will be and its
not constant so i cant use this too.

Any ideas of solving this problem?


Cheers

I realized something very similar to what you described in

http://sourceforge.net/projects/bestgui

- the subprocess2.py module realizes the non blocking I/O
- the outputparser.py module processes the output from the controlled
process and updates the progress-bar, the status-bar and the log
messages in the GUI. Incomplete lines are stored in a buffer and
processed at the next read.

ciao

 

[placid]

Use a thread that reads one character at a time; when it sees
whatever signals "end of line" (it sounds like you're reading a progress
bar implemented via <cr>overwrite). Combine the characters into a
string, return the string to the main program via a queue.


If there is no such "end of line" character, but there IS a
noticeable delay between "writes", a more complex method might suffice
-- in which one thread does the byte reads, setting a time value on each
read; a related thread then does a sleep() loop, checking the "last read
time" against the pause length -- if close enough to the pause duration,
combine and return...

i dont think there is a noticeable delay between "writes".

Alternatively, take a good old style terminal keyboard (a VT100
Tempest-rated model should be ideal), and use it to beat Bill Gates over
the head until he agrees to push a high-priority upgrade to the command
line I/O system... or makes files work with select() (so you can combine
the time-out with the byte read)

Tsk Tsk

 

[Dennis Lee Bieber]

Tsk Tsk

Have you ever seen a Tempest VT-100? Lead shielding on the monitor
FACE... Turn the brightness all the way up and it still looked dim.
Fiber optic to the computer. And a shield keyboard about three inches
thick and weighing 5 lbs.

All to keep "them" from using radio equipment to pick up key strokes
or the display scan line radiation (use an external vertical/horizontal
sweep generator to an oscilloscope, and a high-gain amplifier to detect
the brightness variation of the CRT).
--
Wulfraed Dennis Lee Bieber KD6MOG
(e-mail address removed) (e-mail address removed)
HTTP://wlfraed.home.netcom.com/
(Bestiaria Support Staff: (e-mail address removed))
HTTP://www.bestiaria.com/

 

[placid]

Have you ever seen a Tempest VT-100? Lead shielding on the monitor
FACE... Turn the brightness all the way up and it still looked dim.
Fiber optic to the computer. And a shield keyboard about three inches
thick and weighing 5 lbs.

All to keep "them" from using radio equipment to pick up key strokes
or the display scan line radiation (use an external vertical/horizontal
sweep generator to an oscilloscope, and a high-gain amplifier to detect
the brightness variation of the CRT).

Nope, i have not seen one.

 

[Paul Du Bois]

What i need to do is, create a process using subprocess.Popen, where
the subprocess outputs information on one line (but the info
continuesly changes and its always on the same line) and read this
information without blocking, so i can retrieve other data from the
line i read in then put this in a GUI interface.

readline() blocks until the newline character is read, but when i use
read(X) where X is a number of bytes then it doesnt block(expected
functionality) but i dont know how many bytes the line will be and its
not constant so i cant use this too.

I wrote something for this the other day. The class has a getline()
method, which either returns a line or raises an exception instead of
blocking. It also raises an exception instead of returning EOF.

My use case had me reading from multiple processes at once; since
select() doesn't work on files in win32, I had to get a little cheesy.
I've appended the function that implements that use case, for
reference.

The central idea is to use PeekNamedPipe to figure out what's in the
pipe. You can then read that data without fear of blocking. I wrote it
quickly, therefore the code is a little embarassing, but... hopefully
useful all the same.

.. class NoLineError(Exception): pass
.. class NoMoreLineError(Exception): pass
.. class liner(object):
.. """Helper class for multi_readlines."""
.. def __init__(self, f):
.. self.fd = f.fileno()
.. self.osf = msvcrt.get_osfhandle(self.fd)
.. self.buf = ''
..
.. def getline(self):
.. """Returns a line of text, or raises NoLineError, or
NoMoreLineError."""
.. try:
.. data, avail, _ = win32pipe.PeekNamedPipe(self.osf, 0)
.. except pywintypes.error:
.. # Pipe closed: give up what we have, then that's it
.. if self.buf:
.. ret, self.buf = self.buf, None
.. return ret
.. else:
.. raise NoMoreLineError
.. if avail:
.. self.buf += os.read(self.fd, avail)
..
.. idx = self.buf.find('\n')
.. if idx >= 0:
.. ret, self.buf = self.buf[:idx+1], self.buf[idx+1:]
.. return ret
.. else:
.. raise NoLineError
..
..
.. def multi_readlines(fs):
.. """Read lines from |fs|, a list of file objects.
.. The lines come out in arbitrary order, depending on which files
.. have output available first."""
.. if type(fs) not in (list, tuple):
.. raise Exception("argument must be a list.")
.. objs = [liner(f) for f in fs]
.. for i,obj in enumerate(objs): obj._index = i
.. while objs:
.. for i,obj in enumerate(objs):
.. try:
.. yield (obj._index, obj.getline())
.. except NoLineError:
.. pass
.. except NoMoreLineError:
.. del objs
.. break # Because we mutated the array

 

[Durumdara]

Hi !

Sorry, but I want to share my experiences. I hope this help to you.

I think that specialized MSWindows based services too complicated. They have
to many bug possibilites.
So I trying with normal, "in python accessable" pipes. I see that with
flush(), and some of the bintotext tricks I can use the
subprocess/masterprocess communication.
Ok, this is not asynchronous. I can send some job to sp(s), and I can
receive the report from sp(s).
But with threading I can create non-blocking communication.

You can see in the example: the PipeBPPThr define a pipe based process-pool
thread.
This can communicate with a subprocess, can send/receive jobs, etc.

If you collect these threads, and write a process pool object, you can
handle all of the communications with one object.

I hope to these examples can help to you.

If not, you can try with wm_copydata messages in Windows.
http://msdn.microsoft.com/library/d...rface/dataexchange/datacopy/usingdatacopy.asp
This way of data exchanging is based on the message handling/sending.

dd