Nonblocking IO read

[srobertjames]

How can I perform a nonblocking IO read? That is, read whatever is
there to read and return. If there's nothing yet, just return.

Failing that, is there a way to manually check if there's anything to
read or the read will block?

(Note that I'm developing for both 'nix and Windows.)

 

[ara.t.howard]

How can I perform a nonblocking IO read? That is, read whatever is
there to read and return. If there's nothing yet, just return.

Failing that, is there a way to manually check if there's anything to
read or the read will block?

(Note that I'm developing for both 'nix and Windows.)

it can't be done. search the archives, this sort of thing almost always
indicates a design flaw. for instance - what will your program do if there is
no input?

regards.

-a

 

[Tom Pollard]

That's what the Unix read() function does ("man 2 read"), available
in Ruby as IO#sysread(). The ri documentation doesn't explain much
about how to use Ruby's sysread, unfortunately.

The select() function (Kernel#select, in Ruby) is the standard way to
do IO multiplexing under Unix. You give it a list of file handles on
which you're waiting to be able to read or write, and it returns when
one or more of them is ready. Windows provides a select() function,
too, but it only supports sockets (not pipes or ordinary file IO.)
The Win32 API generally doesn't provide non-blocking IO methods,
because it's assumed you'll use threads if you want to do something
else while waiting for IO.

it can't be done. search the archives, this sort of thing almost
always
indicates a design flaw. for instance - what will your program do
if there is
no input?

Check other IO channels, wait for other sorts of events, do work -
whatever. Where do come from asserting that the use of non-blocking
IO represents a "design flaw"?

Tom

 

[srobertjames]

it can't be done. search the archives, this sort of thing almost always
indicates a design flaw. for instance - what will your program do if there is
no input?

I'm running an external command (using popen4 - yes, it is available on
Windows also). I capture stdout and stderr. "what will your program do
if there is no input?" - I assume you mean output - Well, I hope there
is no stderr output. If there is, I'd like to capture it and feed it
to Logger.

Likewise, if there is a problem, and there's no stdout, then I don't
want to wait for ever - I want to throw an exception.

 

[srobertjames]

In the interests of promoting creativity, I'll ask the question without
any prior assumptions:

I'm using popen4 (great gem, btw) to run an external command and
capture stdout and stderr. If one (or both) of those is empty, I don't
want to hang. Just keep on moving. In fact, stderr will normally be
empty.

How could I do this?

(I'm supporting 'nix and Windows)

 

[ara.t.howard]

That's what the Unix read() function does ("man 2 read"), available in Ruby
as IO#sysread(). The ri documentation doesn't explain much about how to use
Ruby's sysread, unfortunately.

no, that blocks. try this:

harp:~ > ruby -e 'STDIN.sysread(1)'

it blocks. the OP wants a non-blocking call. eg

harp:~ > ruby -r io/nonblock -e 'STDIN.nonblock{ STDIN.sysread(1) }'
-e:1:in `sysread': Resource temporarily unavailable (Errno::EAGAIN)
from -e:1
from /home/ahoward//lib/ruby/1.8/io/nonblock.rb:19:in `nonblock'
from -e:1

The select() function (Kernel#select, in Ruby) is the standard way to do IO
multiplexing under Unix. You give it a list of file handles on which you're
waiting to be able to read or write, and it returns when one or more of them
is ready. Windows provides a select() function, too, but it only supports
sockets (not pipes or ordinary file IO.) The Win32 API generally doesn't
provide non-blocking IO methods, because it's assumed you'll use threads if
you want to do something else while waiting for IO.

right. but ruby's thread are green and, on windows, block the entire process
when waiting on IO so not an option here.

Check other IO channels, wait for other sorts of events, do work - whatever.
Where do come from asserting that the use of non-blocking IO represents a
"design flaw"?

because nine times out of ten code has nothing to do if no input is available.
if it does that's another story, but then there are basically two options:

- use select or an event abstraction in one giant loop. if you do this you are
going to block anyhow, only based on all inputs, not just one.

- use threads, only you can't if you're wanting ruby code to run nb in windows.


i'm not saying there is never a case for nbio - i use it myself - just that,
unless a developer has a good reason to be doing something else it's not
required and, as a search of ruby-talk will show, that is often the case.

regards.

-a

 

[ara.t.howard]

I'm running an external command (using popen4 - yes, it is available on
Windows also). I capture stdout and stderr. "what will your program do if
there is no input?" - I assume you mean output - Well, I hope there is no
stderr output. If there is, I'd like to capture it and feed it to Logger.

Likewise, if there is a problem, and there's no stdout, then I don't want to
wait for ever - I want to throw an exception.

so, basically you want to timeout on a read right? in unix you can just use

require 'timeout'

Timeout::timeout(n) do
Open4.popen4 ...
...
...
end

but, this is implimented via a thread so i think the read is going to block
your entire process in windows... you might try it.

-a

 

[S. Robert James]

Not very attractive...

Back to the original question: there is no simple way to read
nonblocking, or see if any output is available ?!?!

 

[Ryan Davis]

Not very attractive...

Back to the original question: there is no simple way to read
nonblocking, or see if any output is available ?!?!

in io/wait:

--- IO#ready?
returns non-nil if input available without blocking, or nil.

also: ri IO.read_nonblock

 

[Bjorn Borud]

[[email protected]]
|
| it can't be done. search the archives, this sort of thing almost
| always indicates a design flaw. for instance - what will your
| program do if there is no input?

nonblocking IO is not a design-flaw; it is something that is in common
use and is supported by most languages with a decent socket API. on
some OS'es this is even available for file IO (although widespread
language support seems to lag behind).

Java for example got proper nonblocking socket IO in 1.4 (see
JSR-000051 "New I/O APIs for the JavaTM Platform").

for examples of use you might want to check out the Reactor pattern
and other patterns for concurrent programming.

-Bjørn

 

[Tom Pollard]

In the interests of promoting creativity, I'll ask the question
without
any prior assumptions:

I'm using popen4 (great gem, btw) to run an external command and
capture stdout and stderr. If one (or both) of those is empty, I
don't
want to hang. Just keep on moving. In fact, stderr will normally be
empty.

How could I do this?

Have you tried simply reading from them? When the command
terminates, that should close those pipes on the command's end. At
that point, a simple read on the stdout handle, for instance, will
return whatever output there was and then eof. If there's no output,
the read should return immediately. Did you try that yet? Did you
try the (Windows-oriented) example from the 'ri Open4' page? If
you're working in Unix you could try

status = POpen4:oen4('cat -') do |stdout, stderr, stdin|
stdin.puts 'hello world'
stdin.close
puts "stdout: #{stdout.read.strip}"
puts "stderr: #{stderr.read.strip}"
end

Anyway, you would only need nonblocking IO if you wanted to read bits
of the stderr stream before the command exited, but that doesn't
sound like what you're want.

Tom

 

[Bjorn Borud]

[Tom Pollard <[email protected]>]
|
| The select() function (Kernel#select, in Ruby) is the standard way to
| do IO multiplexing under Unix. You give it a list of file handles on
| which you're waiting to be able to read or write, and it returns when
| one or more of them is ready. Windows provides a select() function,
| too, but it only supports sockets (not pipes or ordinary file IO.)

if memory serves, a select(2) on a file descriptors representing a
(disk) file only had defined semantics when used to wait on mandatory
locking and is not used for ensuring that a read(2) or write(2) will
not block. (I might be in error since it has been a while since I
looked into it).

-Bjørn

 

[Tom Pollard]

If you're working in Unix you could try

status = POpen4:oen4('cat -') do |stdout, stderr, stdin|

D'oh! (typo) That should have been

status = POpen4:open4('cat -') do |stdout, stderr, stdin|
stdin.puts 'hello world'
stdin.close
puts "stdout: #{stdout.read.strip}"
puts "stderr: #{stderr.read.strip}"
end


Tom

 

[John Carter]

The select() function (Kernel#select, in Ruby) is the standard way to do IO
multiplexing under Unix.

At a very deep level (last time I looked) thread animation and IO met
and became the same thing in a giant select lurking in the core of the
ruby Interpreter.

ie. Use select / spin another thread amounted cosmically to the same
thing. Choose whatever is easiest for you.




John Carter Phone : (64)(3) 358 6639
Tait Electronics Fax : (64)(3) 359 4632
PO Box 1645 Christchurch Email : (e-mail address removed)
New Zealand

"We have more to fear from
The Bungling of the Incompetent
Than from the Machinations of the Wicked." (source unknown)

 

[ara.t.howard]

[[email protected]]
|
| it can't be done. search the archives, this sort of thing almost
| always indicates a design flaw. for instance - what will your
| program do if there is no input?

nonblocking IO is not a design-flaw; it is something that is in common
use and is supported by most languages with a decent socket API. on
some OS'es this is even available for file IO (although widespread
language support seems to lag behind).

Java for example got proper nonblocking socket IO in 1.4 (see
JSR-000051 "New I/O APIs for the JavaTM Platform").

but it's archaic compared to event driven frameworks like

http://www.monkey.org/~provos/libevent/
http://liboop.ofb.net/

in addition it's a hack in the c lib of many oses.

for examples of use you might want to check out the Reactor pattern
and other patterns for concurrent programming.

afaik the reactor pattern is a synchronous pattern

http://www.artima.com/articles/io_design_patterns2.html

not unlike the model of libevent and liboop - which are both synchronous...
am i missing something?

-a

 

[ara.t.howard]

Ara, I'm curious to know why you think the usage of nbio is archaic or
indicative of a design flaw. I wrote the EventMachine library in order to
make certain kinds of Ruby programs easier to write, and it uses nbio
pervasively. I'm fairly sure libevent does, also. EM works with almost any
kind of descriptor, except for certain things on Windows (like _pipe()
descriptors, which are nonselectable, and some of the native Windows objects
that don't support a socket-like API).

Up until recently (late May 2006), Ruby didn't have complete support for
nbio on all descriptor types, which is one of the reasons that EventMachine
includes a compiled C++ extension. EM interoperates perfectly well with Ruby
threads,* which is another important reason to use nbio.

hi francis-

it's exactly things like eventmachine that make me say using nbio is archaic -
i don't need to handle the complexities of nbio when powerful abstractions
like it exist!

for instance, consider the OP's orignial question: basically he wanted to
timeout if stdout was not produced in a certain amount of time. i certainly
wouldn't delve into the complexities of nbio to handle this, but instaed might
do something along the lines of

harrp:~ > cat a.rb
require 'open4' and require 'timeout'

def might_take_too_long cmd, stdin, timeout = 42
stdout, stderr = '', ''
Timeout::timeout(timeout) do
open4.popen4(cmd) do |cid,i,o,e|
i.write stdin and i.close
o.each{|line| stdout << line} and e.each{|line| stderr << line}
end
end
[ stdout, stderr ]
end

p might_take_too_long('ruby', 'sleep 1 and p 42', 2)
p might_take_too_long('ruby', 'sleep 42 and p 42', 2)


harp:~ > ruby a.rb
["42\n", ""]
/home/ahoward//lib/ruby/1.8/timeout.rb:43:in `might_take_too_long': execution expired (Timeout::Error)
<snip>


now, timeout might use any manner of select and/or nbio - i don't care. my
point is just that, for 90-99% of the problems seem to want to solve with
nbio, it is way too low level a mechanism to skin the cat and prefer to stand
on the shoulders of powerful abstractions, like eventmachine or threads or
whatever, and avoid the painful details of EAGAIN, race conditions (it wasn't
ready so i... oops, now it is ready), and stdio buffers being lost.

for the record, i have code which uses nbio, io/nonblock, etc. it's simply my
observation that a majority of the posts to this list about nbio have two
issues:

- nothing else can really be done while waiting for io. eg. nbio isn't
needed for the problem at hand

- something else could simoultaneously be done, but the problem could be
more elegantly solved with threads, queues, readpartial, or something even
more abstract

the rest, of course, really do need nbio. a quick search of archives though
is an eye opener into the difficulties of mixing nbio and stdio, check out
these threads, specifically the posts by tanaka akira, who knows about 10
billion times more about nbio and ruby than i do

http://groups-beta.google.com/group...gst&q=non+blocking+io&rnum=1#23e22cc9e56a3200

http://groups-beta.google.com/group...gst&q=non+blocking+io&rnum=6#775af5d9900aa716

http://groups-beta.google.com/group...gst&q=non+blocking+io&rnum=6#775af5d9900aa716

http://groups-beta.google.com/group...gst&q=non+blocking+io&rnum=6#775af5d9900aa716


his insight is enough to raise 'too low-level' warnings in my mind! at least
until using nbio from a ruby script doesn't require such deep understanding of
ruby's internals

kind regards.

-a

 

[Robert Klemme]

Anyway, you would only need nonblocking IO if you wanted to read bits of
the stderr stream before the command exited, but that doesn't sound like
what you're want.

Actually this is not correct: if there is a lot written to stderr then
you need to read that concurrently. If you do not do that then the
process will block on some stderr write operation that fills up the pipe
and you get a deadlock because your code waits for process termination.

Kind regards

robert

 

[Bill Kelly]

In the interests of promoting creativity, I'll ask the question without
any prior assumptions:

I'm using popen4 (great gem, btw) to run an external command and
capture stdout and stderr. If one (or both) of those is empty, I don't
want to hang. Just keep on moving. In fact, stderr will normally be
empty.

How could I do this?

(I'm supporting 'nix and Windows)

Windows will be a problem. Admittedly, I haven't tried
ruby 1.8.5 yet, which has new nonblock_* methods. However,
my expectation is that you'll only get nonblocking behavior
on windows from sockets, not from pipes.

On Windows, calling select() on a pipe, always returns
immediately with "data ready to read", regardless if there's
any data there or not.

This has been the bane of my existence on Windows ruby for
5 or 6 years. I do IPC on Windows ruby using TCP over
loopback, instead of pipes, in order to get nonblocking
semantics. (That still doesn't help for reading from the
console, though... (search the archives for 'kbhit' for a
partial solution there...))

One of these years, I'd like to chat with a Windows guru
and ask how he/she would recommend making a select() that
works on both sockets and pipes on Windows. Ruby could
*really* use one.


Regards,

Bill

 

[Gabriele Marrone]

Il giorno 01/nov/06, alle ore 01:33, (e-mail address removed) ha scritto:

right. but ruby's thread are green and, on windows, block the
entire process
when waiting on IO so not an option here.

I thought Ruby internally uses non-blocking I/O in order to avoid
that a green thread reading something blocks every other thread: am I
wrong?
Or is this true just under unix?

 

[Bill Kelly]

Il giorno 01/nov/06, alle ore 01:33, (e-mail address removed) ha scritto:


I thought Ruby internally uses non-blocking I/O in order to avoid
that a green thread reading something blocks every other thread: am I
wrong?
Or is this true just under unix?

Ruby uses select() internally, and Windows doesn't support
select() on pipes, just sockets.


Regards,

Bill