[ANN] socket_sendfile

[Eric Hodel]

= socket_sendfile

Rubyforge Project:

http://rubyforge.org/projects/rctools/

Documentation:

http://dev.robotcoop.com/Libraries/socket_sendfile/

== About

Socket#sendfile implements sendfile(2) for sending files without
copying data
to the process. See the sendfile(2) manual page for more details.

Note that your system must support the sendfile(2) system call the
same way
FreeBSD does for Socket#sendfile to work.

== Installing socket_sendfile

First you need an OS that supports sendfile() the way FreeBSD does.

Then install the gem:

$ sudo gem install socket_sendfile

== Using socket_sendfile

require 'rubygems'
require 'socket'
require 'socket_sendfile'

socket = TCPSocket.open host, port

File.open 'myfile' do |fp|
socket.sendfile fp
end

 

[Stephen Bannasch]

[partially answering my own question ...]

The reason that builder can't create xml elements with "-" chars in
the element is because strings with "-" chars are not legal Ruby
symbols. Builder uses method_missing to create the xml element markup
and so it seems Ruby's parser chokes when it tries to convert the
string into a symbol.

I've got a workaround below but it is more general than I want (it
replaces all "_" chars with "-"). I left some code commented out that
I was trying too get working but didn't. It's a bit confusing to
debug because the inheritance chain (XmlMarkup < XmlBase <
BlankSlate) end up hiding all the normal methods -- I'm a bit
surprised that my code works at all.

Anyway, I'm just learning Ruby and figuring out just what is going on
here is fun. If anybody has suggestions I'd appreciate it.

Thanks

#!/usr/bin/env ruby
# file: jnlpmarkup.rb

# extends Builder::XmlMarkup for creating XML markup for jnlp documents.
# Some jnlp elements have a "-" char in the element name. Builder doesn't
# accept this character. So this class postprocesses the result and replaces
# proxy jnlp element names such as "<application_desc>" with
"<application-desc>"
#
# because I can't get what I intended to work right now this just
# replaces all "_" chars with '-' chars

require 'builder/xmlmarkup'

module Builder

class JnlpMarkup < XmlMarkup

@@jnlp_in = %w{application_desc applet_desc component_desc
installer_desc offline_allowed related_content all_permissions
j2ee_application_client_permissions ext_download}
@@jnlp_out = %w{application-desc applet-desc component-desc
installer-desc offline-allowed related-content all-permissions
j2ee-application-client-permissions ext-download}

def method_missing(sym, *args, &block)
super.gsub(/_/, '-')

# various things that didn't work below:
# x = super
# x = x.gsub(/application_desc/, 'application-desc')
# x = x.gsub(/applet_desc/, 'applet_desc')
# x = x.gsub(/#{@@jnlp_in}/, @@jnlp_out)
# @@jnlp_in.each_index do |i| x = x.gsub(/#{@@jnlp_in}/,
@@jnlp_out) end
end
end
end

 

[ara.t.howard]

= socket_sendfile

Rubyforge Project:

http://rubyforge.org/projects/rctools/

Documentation:

http://dev.robotcoop.com/Libraries/socket_sendfile/

== About

Socket#sendfile implements sendfile(2) for sending files without copying data
to the process. See the sendfile(2) manual page for more details.

Note that your system must support the sendfile(2) system call the same way
FreeBSD does for Socket#sendfile to work.

== Installing socket_sendfile

First you need an OS that supports sendfile() the way FreeBSD does.

Then install the gem:

$ sudo gem install socket_sendfile

== Using socket_sendfile

require 'rubygems'
require 'socket'
require 'socket_sendfile'

socket = TCPSocket.open host, port

File.open 'myfile' do |fp|
socket.sendfile fp
end

i'm just about to write a server which sends 1-3gb images as the response...
this could be quite useful.

one gripe. it doesn't work:

/home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group readable by
default. any way to adjust this? an env var perhaps? seems like this should
just warn.


ok. one more issue:

fortytwo :~/tmp > ruby ../a.rb
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:392: warning: Insecure world writable dir /usr/local, mode 040777
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb: In function `bsock_sendfile':
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:46: warning: implicit declaration of function `sendfil
e'
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:396:in `build': error executing gcc -shared -Wall -W -Wpointer-arith -
Wcast-qual -Wcast-align -Wwrite-strings -Wmissing-noreturn -Werror -g -O2 -I /usr/local/ruby-1.8.4/lib/ruby/1.8/i686-linux -o /home/ahoward/.rub
y_inline/Inline_BasicSocket_0a2b.so /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c : 256 (CompilationError)
Renamed /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c to /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c.bad from /usr/local/
ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:591:in `inline'
from /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/socket_sendfile-1.1.0/lib/socket_sendfile.rb:6
from /usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:27:in `require'
from ../a.rb:3


my system definitely has sendfile


fortytwo :~/tmp > PAGER=cat man sendfile|head -12
SENDFILE(2) Linux Programmer's Manual SENDFILE(2)



NAME
sendfile - transfer data between file descriptors

SYNOPSIS
#include <sys/sendfile.h>

ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);


guess the signature won't cut it though eh? maybe i'll just have to use dl?

regards.

-a

 

[Eric Hodel]

--Apple-Mail-4--479933733
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed

i'm just about to write a server which sends 1-3gb images as the
response...
this could be quite useful.

one gripe. it doesn't work:

/home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group
readable by
default. any way to adjust this? an env var perhaps? seems like
this should
just warn.

That's RubyInline complaining. Let me check with Ryan Davis to see
what can be done.

ok. one more issue:

fortytwo :~/tmp > ruby ../a.rb
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./
inline.rb:392: warning: Insecure world writable dir /usr/local,
mode 040777
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/
socket_sendfile-1.1.0/lib/socket_sendfile.rb: In function
`bsock_sendfile':
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/
socket_sendfile-1.1.0/lib/socket_sendfile.rb:46: warning: implicit
declaration of function `sendfil
e'
/usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./
inline.rb:396:in `build': error executing gcc -shared -Wall -W -
Wpointer-arith -
Wcast-qual -Wcast-align -Wwrite-strings -Wmissing-noreturn -
Werror -g -O2 -I /usr/local/ruby-1.8.4/lib/ruby/1.8/i686-linux -o /
home/ahoward/.rub
y_inline/Inline_BasicSocket_0a2b.so /home/ahoward/.ruby_inline/
Inline_BasicSocket_0a2b.c : 256 (CompilationError)
Renamed /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c
to /home/ahoward/.ruby_inline/Inline_BasicSocket_0a2b.c.bad
from /usr/local/
ruby-1.8.4/lib/ruby/gems/1.8/gems/RubyInline-3.4.0/./inline.rb:
591:in `inline'
from /usr/local/ruby-1.8.4/lib/ruby/gems/1.8/gems/
socket_sendfile-1.1.0/lib/socket_sendfile.rb:6
from /usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8/
rubygems/custom_require.rb:27:in `require'
from ../a.rb:3


my system definitely has sendfile

#include <sys/sendfile.h>

ssize_t sendfile(int out_fd, int in_fd, off_t *offset,
size_t count);


guess the signature won't cut it though eh? maybe i'll just have
to use dl?

On FreeBSD it looks like this:

#include <sys/types.h>
#include <sys/socket.h>
#include <sys/uio.h>

int
sendfile(int fd, int s, off_t offset, size_t nbytes,
struct sf_hdtr *hdtr, off_t *sbytes, int flags);

And has this to say about non-blocking sockets:

When using a socket marked for non-blocking I/O, sendfile() may
send
fewer bytes than requested. In this case, the number of bytes
success-
fully written is returned in *sbytes (if specified), and the
error EAGAIN
is returned.

Try this patch:


--Apple-Mail-4--479933733
Content-Transfer-Encoding: 7bit
Content-Type: application/octet-stream;
x-unix-mode=0644;
name="socket_sendfile.rb.linux.patch"
Content-Disposition: attachment;
filename=socket_sendfile.rb.linux.patch

Index: lib/socket_sendfile.rb
===================================================================
--- lib/socket_sendfile.rb (revision 8257)
+++ lib/socket_sendfile.rb (working copy)
@@ -4,9 +4,7 @@
class BasicSocket

inline do |builder|
- builder.include '<sys/types.h>'
- builder.include '<sys/socket.h>'
- builder.include '<sys/uio.h>'
+ builder.include '<sys/sendfile.h>'

builder.include '"ruby.h"'
builder.include '"rubyio.h"'
@@ -25,14 +23,9 @@
static VALUE bsock_sendfile(int argc, VALUE *argv, VALUE sock) {
OpenFile *fptr, *sptr;
VALUE io;
- int fd, s, res;
- off_t sent_bytes = 0;
- size_t sent_total = 0;
- struct timeval timeout;
+ int fd, s;
+ ssize_t sent_bytes;

- timeout.tv_sec = 0;
- timeout.tv_usec = 500;
-
rb_scan_args(argc, argv, "10", &io);

GetOpenFile(io, fptr);
@@ -41,23 +34,10 @@
GetOpenFile(sock, sptr);
s = fileno(sptr->f);

- for (; {
- TRAP_BEG;
- res = sendfile(fd, s, sent_total, 0, NULL, &sent_bytes, 0);
- TRAP_END;
+ TRAP_BEG;
+ sent_bytes = sendfile(fd, s, 0, 0);
+ TRAP_END;

- sent_total += sent_bytes;
-
- if (res == 0)
- break;
-
- if (res < 0 && errno != EAGAIN)
- rb_sys_fail("sendfile(2)");
-
- // This socket is full, let's see if we can do work elsewhere.
- rb_thread_select(0, NULL, NULL, NULL, &timeout);
- }
-
return INT2FIX(sent_total);
}
EOC

--Apple-Mail-4--479933733
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
charset=US-ASCII;
delsp=yes;
format=flowed


If that doesn't work try an fstat on the fd and set the count to
stat.st_size. Rumor has it that Linux might require an explicit count.

--
Eric Hodel - (e-mail address removed) - http://blog.segment7.net
This implementation is HODEL-HASH-9600 compliant

http://trackmap.robotcoop.com



--Apple-Mail-4--479933733--

 

[Eric Hodel]

i'm just about to write a server which sends 1-3gb images as the
response...
this could be quite useful.

one gripe. it doesn't work:

/home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group
readable by
default.

Your home dir is group and world writable. A 755 directory works,
the message is broken. (Well, just fixed in perforce.)

any way to adjust this? an env var perhaps?

The INLINEDIR environment variable controls this, so set it to a
directory that's got the right permissions (at least 755).

seems like this should just warn.

Other people may be able to inject code into ~/.ruby_inline if the
directory is writeable by other people than you.

That would be Really Bad.

A warning is insufficient. You would be royally screwed by the time
you read it.

 

[Ryan Davis]

i'm just about to write a server which sends 1-3gb images as the
response...
this could be quite useful.

one gripe. it doesn't work:

/home/ahoward is insecure (40777), needs 0700 for perms. Exiting

i work in a collaborative lab... all our home dirs are group
readable by
default. any way to adjust this? an env var perhaps? seems like
this should
just warn.

Your home dir is world and group writable, not just group readable.
There is a big big big difference between those two. There is no way
I'm going to let it just warn in that case. However, you do point out
that the error message poor. I've changed the error message to:

"#{path} is insecure (#{'%o' % mode}). It may not be group or
world writable. Exiting."

 

[ara.t.howard]

Your home dir is world and group writable, not just group readable. There is
a big big big difference between those two. There is no way I'm going to let
it just warn in that case. However, you do point out that the error message
poor. I've changed the error message to:

"#{path} is insecure (#{'%o' % mode}). It may not be group or world
writable. Exiting."

i totally understand what you are saying. however i dont' see why a ruby lib
should be more secure that ruby itself:

fortytwo :/tmp > ls -ltar /|grep tmp
drwxrwxrwt 21 root root 815104 Mar 24 07:41 tmp

fortytwo :/tmp > echo 'puts 42' > a.rb
fortytwo :/tmp > chmod 777 a.rb

fortytwo :/tmp > ruby -W2 -e' $SAFE = 0; load "a.rb"'
42

it sure seems fine loading world writable files from world writable
directories... unless you set $SAFE above the default:

fortytwo :/tmp > ruby -W2 -e' $SAFE = 1; load "a.rb"'
-e:1: warning: Insecure world writable dir /usr/local, mode 040777
-e:1:in `load': loading from unsafe path /usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8:/usr/local/ruby-1.8.4/lib/ruby/site_ruby/1.8/i686-linux:/usr/local/ruby-1.8.4/lib/ruby/site_ruby:/usr/local/ruby-1.8.4/lib/ruby/1.8:/usr/local/ruby-1.8.4/lib/ruby/1.8/i686-linux:. (SecurityError)
from -e:1


this seems sufficient and allows stupid people like me to shoot their own foot
off if they choose. does RubyInline really need more that this : do nothing
unless not $SAFE.nil? and $SAFE > 0?

regards.

-a

 

[ara.t.howard]

Your home dir is group and world writable. A 755 directory works, the
message is broken. (Well, just fixed in perforce.)

thanks figured that out later... unfortunately it needs to be as some webdav
stuff lands there and www has it's own group which i cannot affect...

The INLINEDIR environment variable controls this, so set it to a directory
that's got the right permissions (at least 755).

this works - thanks!

Other people may be able to inject code into ~/.ruby_inline if the directory
is writeable by other people than you.

That would be Really Bad.

A warning is insufficient. You would be royally screwed by the time you
read it.

all good but not at $SAFE==0 or $SAFE==nil. after all, $SAFE==nil really
ought to mean $VERY_DANGEROUS_PLEASE==true! ;-)

regards.

-a

 

[ara.t.howard]

Was there something wrong with my existing ruby-sendfile library?

http://rubyforge.org/projects/ruby-sendfile

ahhhhhh.

;-)


-a

 

[Toby DiPasquale]

ahhhhhh.

;-)

Does that work for you, ara? It should work on Linux, Solaris and
FreeBSD out of the box. I didn't put it in a gem because I wasn't sure
how to get the GemSpec to correctly identify only the platforms on which
it would build. Let me know.

 

[ara.t.howard]

Does that work for you, ara? It should work on Linux, Solaris and
FreeBSD out of the box. I didn't put it in a gem because I wasn't sure
how to get the GemSpec to correctly identify only the platforms on which
it would build. Let me know.

it sure as heck does!


harp:~/ruby-sendfile > cat a.rb
require 'sendfile'

open(__FILE__){|f| STDOUT.sendfile f}




harp:~/ruby-sendfile > ruby a.rb
require 'sendfile'

open(__FILE__){|f| STDOUT.sendfile f}


this is fantastic. basically i'll be responding to certain web requests with
something like

send_mime_header
sendfile huge_massive_file

and was thinking i'd have to fork a cat to make it fast enough - so this is
fantastic.

thanks alot.

-a

 

[Eric Hodel]

Was there something wrong with my existing ruby-sendfile library?

http://rubyforge.org/projects/ruby-sendfile

I didn't know it existed, but:

It isn't as easy to install as a gem is.

It doesn't work well with non-blocking sockets on FreeBSD. Handling
it in Ruby is too annoying to be worthwhile.

Mine is 1.0.

Mine has tests.

 

[Toby DiPasquale]

It isn't as easy to install as a gem is.
Given.

It doesn't work well with non-blocking sockets on FreeBSD. Handling
it in Ruby is too annoying to be worthwhile.

Thanks for the first suggestions for improvements to ruby-sendfile. I
will fix it over the weekend and make a gem out of it. That way Linux
and Solaris users won't be left out, as I had originally intended they
not be.

P.S. In your test suite for socket_sendfile, you can use the
IO#nonblock= call by requiring 'io/nonblock' instead of having to
explicitly call fcntl. Makes the code more readable, IMO.