Nonblocking Sockets

[James Edward Gray II]

Is this the "standard" way to make a nonblocking Socket in Ruby?

require "socket"
require "fcntl"

server = TCPServer.new(...)
server.fcntl(Fcntl::F_SETFL, Fcntl::O_NONBLOCK)

__END__

Will that work on Unix and Windows?

If I accept() a connection from that, will the connecting socket be
nonblocking as well?

Thanks for the information.

James Edward Gray II

 

[Bill Kelly]

Is this the "standard" way to make a nonblocking Socket in Ruby?

require "socket"
require "fcntl"

server = TCPServer.new(...)
server.fcntl(Fcntl::F_SETFL, Fcntl::O_NONBLOCK)

__END__

Will that work on Unix and Windows?

As far as I know it's the "standard" way (but that
just means it's the only way I know of.

It doesn't work on Windows.

I've ended up tacking on: ... if defined? Fcntl::O_NONBLOCK
onto the fcntl() line so that my scripts still run on
windows, even though they may block inappropriately.

Blocking I/O issues have been a thorny issue for me in
trying to write cross-platform network applications in
Ruby.

If I accept() a connection from that, will the connecting socket be
nonblocking as well?

I seem to recall discussion of a patch to fix something
related to nonblocking I/O and accept() within the last
year or so... but I could be wrong... (I couldn't find
it with Google.)

I tend to re-issue the nonblock fcntl() call before
every socket operation I perform.

Even so, I *still* hit this timeout occasionally:

if select([sock], nil, nil, UDP_RECV_TIMEOUT)
begin
timeout(UDP_RECV_TIMEOUT) {
sock.fcntl(Fcntl::F_SETFL, Fcntl::O_NONBLOCK) if defined? Fcntl::O_NONBLOCK
resp = sock.recvfrom(65536)
}
rescue Timeout::Error
$stderr.puts "Timeout::Error in sock.recvfrom !"
end
end

...Since select() said data was ready, AND since I'm
requesting a nonblocking operation... I have no idea
why #recvfrom sometimes hangs. It used to totally hang
my program (on linux), indefinitely, about once a day,
until I added the timeout().


Regards,

Bill

 

[nobu.nokada]

Hi,

At Sun, 17 Jul 2005 05:52:34 +0900,
James Edward Gray II wrote in [ruby-talk:148402]:

Is this the "standard" way to make a nonblocking Socket in Ruby?

Though not a "standard" way, there is io/nonblock module in
"rough".

Will that work on Unix and Windows?

Still not on Windows.

If I accept() a connection from that, will the connecting socket be
nonblocking as well?

It may be changed so in the near future.

 

[Tanaka Akira]

I tend to re-issue the nonblock fcntl() call before
every socket operation I perform.

Even so, I *still* hit this timeout occasionally:

if select([sock], nil, nil, UDP_RECV_TIMEOUT)
begin
timeout(UDP_RECV_TIMEOUT) {
sock.fcntl(Fcntl::F_SETFL, Fcntl::O_NONBLOCK) if defined? Fcntl::O_NONBLOCK
resp = sock.recvfrom(65536)
}
rescue Timeout::Error
$stderr.puts "Timeout::Error in sock.recvfrom !"
end
end

...Since select() said data was ready, AND since I'm
requesting a nonblocking operation... I have no idea
why #recvfrom sometimes hangs.

Hmm. Linux, UDP, readable by select, not readable by recvfrom.

It may be caused by wrong UDP checksum.

Linux-Kernel Archive: UDP recvmsg blocks after select(), 2.6 bug?
http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1372.html

Debian Bug report logs - #275585 - /usr/sbin/inetd: UDP builtins can be used to hang inetd
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=275585&archive=yes

select(2):
Under Linux, select may report a socket file descriptor as "ready for
reading", while nevertheless a subsequent read blocks. This could for
example happen when data has arrived but upon examination has wrong
checksum and is discarded. There may be other circumstances. Thus it
may be safer to use O_NONBLOCK on sockets that should not block.

You can test UDP with wrong checksum by hping2.
See the debian bug report #275585.

It used to totally hang
my program (on linux), indefinitely, about once a day,
until I added the timeout().

It seems that Ruby process doesn't hang because timeout works.
timeout is implemented by Ruby thread.

So your problem is IPSocket#recvfrom retry when EAGAIN.

You may need lower level method which makes EAGAIN user visible.

 

[Yohanes Santoso]

server.fcntl(Fcntl::F_SETFL, Fcntl::O_NONBLOCK)

that should work in most cases, but perhaps you should have preserved
any existing values as well, something like:

server.fcntl(Fcntl::F_SETFL, server.fcntl(Fcntl::F_GETFL) |
Fcntl::O_NONBLOCK)

YS.

 

[James Edward Gray II]

It may be changed so in the near future.

So do I need to call it once on each Socket accept() spawns? Will
once be enough?

What is the nonblocking behavior of accept? Will it throw an
Exception, return nil, or what?

Thanks.

James Edward Gray II

 

[nobu.nokada]

Hi,

At Sun, 17 Jul 2005 13:10:03 +0900,
James Edward Gray II wrote in [ruby-talk:148440]:

So do I need to call it once on each Socket accept() spawns? Will
once be enough?

Once on each accepted sockets.

What is the nonblocking behavior of accept? Will it throw an
Exception, return nil, or what?

Errno::EWOULDBLOCK will be raised.

 

[Bill Kelly]

Hmm. Linux, UDP, readable by select, not readable by recvfrom.

It may be caused by wrong UDP checksum.

Linux-Kernel Archive: UDP recvmsg blocks after select(), 2.6 bug?
http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1372.html

Debian Bug report logs - #275585 - /usr/sbin/inetd: UDP builtins can be used to hang inetd
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=275585&archive=yes

Thank you! A most interesting read.

It seems that Ruby process doesn't hang because timeout works.
timeout is implemented by Ruby thread.

So your problem is IPSocket#recvfrom retry when EAGAIN.

You may need lower level method which makes EAGAIN user visible.

Starting with
http://www.ussg.iu.edu/hypermail/linux/kernel/0410.0/1837.html
there were some number of messages in the thread saying
they can't return EAGAIN in this situation because
POSIX forbids it. (But that's in the nonblocking case.)

Since I'm requesting O_NONBLOCK prior to recvfrom(),
shouldn't it ............ ahh "your problem is
IPSocket#recvfrom retry when EAGAIN." Hmm.....

Interesting, doesn't that imply IPSocket#recvfrom is
essentially broken for NONBLOCK semantics? What I mean
is, if the socket is in nonblocking mode, and recvfrom()
gets an EAGAIN, shouldn't it just return nil to the
caller? (Or whatever the TCP methods do in a NONBLOCK
situation?) Why should it ever "block" in a NONBLOCK
situation?


Thanks,

Regards,

Bill

 

[Brian Candler]

As far as I know it's the "standard" way (but that
just means it's the only way I know of.

It doesn't work on Windows.

I've ended up tacking on: ... if defined? Fcntl::O_NONBLOCK
onto the fcntl() line so that my scripts still run on
windows, even though they may block inappropriately.

Blocking I/O issues have been a thorny issue for me in
trying to write cross-platform network applications in
Ruby.

Aside: would it be appropriate to use Ruby threads in your application? It
can hide a lot of socket blocking nastiness. And I presume the interpreter
copes with Windows.

Regards,

Brian.

 

[James Edward Gray II]

Aside: would it be appropriate to use Ruby threads in your
application? It
can hide a lot of socket blocking nastiness. And I presume the
interpreter
copes with Windows.

This works for *some* applications. Unfortunately, I large write can
still stall the process. I wish it wasn't so, but I believe you
still need to go with nonblocking IO for a robust setup.

James Edward Gray II

 

[James Edward Gray II]

Hi,

At Sun, 17 Jul 2005 13:10:03 +0900,
James Edward Gray II wrote in [ruby-talk:148440]:

So do I need to call it once on each Socket accept() spawns? Will
once be enough?

Once on each accepted sockets.

What is the nonblocking behavior of accept? Will it throw an
Exception, return nil, or what?

Errno::EWOULDBLOCK will be raised.

Thanks so much for all this information. You've been very helpful
and I appreciate it.

A couple more questions and then I promise I'm done:

1. Do I then use send() and recvfrom() to write and read data?
(Other messages in this thread have me nervous about recvfrom().)
2. Will these also throw Errno::EWOULDBLOCK?

Thanks again.

James Edward Gray II

 

[Zed A. Shaw]

James,

Looks like this has already been explained from the Ruby perspective,
but thought I'd give you what I know of the "official" information. All
of this comes from Effective TCP/IP Programming by Jon C. Snader, which
is probably the absolute best book on TCP/IP programming ever written.
Go get it or I will hunt you down and paint you up like a clown.

* You should always get the original fcntl settings and reuse them in
the set operation. Many operating systems use fcntl settings in weird
ways.

* You can set O_NONBLOCK on a socket during connect when you want a
client to attempt a connect, and then go back into your select loop to
wait for the response. Also useful as a connect timeout. Add a timeout
to your select and then it will return if the connect..well..times out.
This has the following requirements though:
- If the connection is not established during the connect call, then it
will return an EINPROGRESS error. Not sure what Ruby does with this.
- If the connect is so fast that it happens immediately then you'll get
a 0 (success) return code. You need to check for this after the connect
or else your select will not work. Skip the select if this happens as
you're already connected.
- After the connect comes back you'll need to use your previously saved
fcntl flags to reset it if you don't want nonblocking anymore.

* Nonblocking accept is usually not needed if you're doing a select/poll
style loop. The exception to this is if you have so much processing
that you don't get around to the accept in time to avoid dropping
clients. In this case either re-write your client handlers to not eat
up so much time, or set the accept socket to nonblock so that you can do
multiple accepts in a loop.

* The practical rule on nonblocking accept is:
- An OS could return EWOULDBLOCK, ECONNABORTED, EINTR as a non-fatal
error which just means try again later.
- All other errno are considered bad and you should deal with them as
an error.
- The semantics are different for different platforms, and I believe
Windows is really weird. You'll need to test on as many as you can, but
I think if you handle those you'll be good.

* Select does have a problem with getting the read response right, which
means you almost always need to set the socket to non-blocking, but also
select only indicates that you *can* read/write, not how much. So, if
you need to read/write 1024 bytes, but the OS buffers only allow for 512
at that moment, then your call will block. Finding out how much you can
write is only possible on some OS with non-standard fcntl. It's just
easier to use nonblocking IO. I'm not sure if poll, /dev/poll, epoll,
or kqueue have a similar problem as select.

* The major problem with Windows vs. BSD sockets is that Windows uses a
special SOCKET type rather than an integer file descriptor. Snader's
book has a nice little wrapper around this, but I believe Ruby handles
this problem as well. Where you'll run into problems (as I am with the
libevent extension I'm working on) is when you pass these SOCKETs to an
external library expecting an integer. Still haven't figured that one
out.

Anyway, enjoy.

Zed A. Shaw

On Sun, 2005-07-17 at 05:52 +0900, James Edward Gray II wrote:

 

[Tanaka Akira]

Interesting, doesn't that imply IPSocket#recvfrom is
essentially broken for NONBLOCK semantics? What I mean
is, if the socket is in nonblocking mode, and recvfrom()
gets an EAGAIN, shouldn't it just return nil to the
caller? (Or whatever the TCP methods do in a NONBLOCK
situation?) Why should it ever "block" in a NONBLOCK
situation?

Ruby's nonblocking behaviour is not well designed.
So it should be redesigned anyway.

However nonblocking I/O is used for several reasons.

There is a case that it is fine that an I/O method hides
nonblockingness. For example, nonblocking I/O which is used to avoid
process blocking by write(2) is such case.

Assume a threaded (non-forking) network server program which use
IO#read to read a request and IO#write to write a response and it
doesn't use nonblocking I/O until someone reports DoS problem because
IO#write may block the server process instead of the thread which
invokes IO#write. If some I/O methods behaves differently between
blocking mode and nonblocking mode, it makes hard to fix the DoS
problem: simply set an socket to nonblocking mode by fcntl may not
work well because nonblocking method behavior may not appropriate for
the server program. So all I/O invocation must be examined or
nonblocking mode must be set only around IO#write method. Note that
IO#read in Ruby 1.8 behaves differently between the modes.

So I think it is reasonable that I/O methods hides nonblocking
behavior: retrying when EAGAIN and partial result. Current Ruby
doesn't do it well, though.

I guess there is a case which needs nonblocking behaviour visible from
script. I'm not sure good design to proviede nonblocking behaviour
for such script, though.

 

[Tanaka Akira]

This works for *some* applications. Unfortunately, I large write can
still stall the process. I wish it wasn't so, but I believe you
still need to go with nonblocking IO for a robust setup.

Ruby threads and nonblocking I/O are not exclusive.

How about using the threads with nonblocking I/O?
It should avoid process wide blocking.

 

[Tanaka Akira]

* Select does have a problem with getting the read response right, which
means you almost always need to set the socket to non-blocking, but also
select only indicates that you *can* read/write, not how much. So, if
you need to read/write 1024 bytes, but the OS buffers only allow for 512
at that moment, then your call will block. Finding out how much you can

read 1024bytes doesn't block when OS has 512bytes.