Web services and larger files

[Luke Kanies]

Hi all,

This is an addendum to my earlier thread about web services and ruby,
wherein I was forced to accept the inevitability of using threads.

I've successfully got my client and server talking to each other for
small tasks, but I'm trying to expand to being able to pass (sometimes
large) files around and I've run into what seems almost like a
fundamental problem.

Basically, I'm trying to send a file over the wire, and for files above
about 20k in size I get an indefinite hang.

I've tried this with soap4r and webrick (I initially thought it was a
problem passing binary around, because the binary files I was testing
were the bigger ones), and I just implemented the same functionality in
xmlrpc and its own builtin server. In both cases, I get basically the
same behaviour: Smaller files go over just fine, larger files hang
indefinitely on a 'write' call:

write(4, "<?xml version=\"1.0\" ?><methodCal"..., 1389384

I've verified that I'm depending on essentially none of the same
libraries in my two implementations (other than builtin classes, of
course), yet I get the same behaviour in both cases.

Anyone have any ideas?

I'm currently actually passing the file contents as an argument to the
method call; should I instead somehow attach the file somehow? Is there
an accepted way to pass files around in XML that I don't know about?

Any help would be greatly appreciated.

 

[Luke Kanies]

Sorry, forgot the code.

Here's the actual code:

http://reductivelabs.com/svn/puppet/language/trunk/lib/puppet/filebucket.rb

And the test script:

http://reductivelabs.com/svn/puppet/language/trunk/test/bucket/tc_bucket.rb

 

[Kirk Haines]

I've successfully got my client and server talking to each other for
small tasks, but I'm trying to expand to being able to pass (sometimes
large) files around and I've run into what seems almost like a
fundamental problem.

Boy. I'm not sure if I am going to be able to help you, but I'll take a look
at the code you posted links to in your followup message.

However, I did want to pipe up and say that I have code that is regularly
being used to pass files over the wire as large as 100 megabytes, with no
problems. This is a web based application, and works just fine in three
different scenarios that I have tested:

1) Webrick -> application thread
2) Apache/mod_ruby -> seperate ruby process
3) Lighttpd -> FCGI proxy (written in Ruby) -> seperate ruby process


Kirk Haines

 

[Luke Kanies]

Boy. I'm not sure if I am going to be able to help you, but I'll take a look
at the code you posted links to in your followup message.

I would definitely appreciate it.

However, I did want to pipe up and say that I have code that is regularly
being used to pass files over the wire as large as 100 megabytes, with no
problems. This is a web based application, and works just fine in three
different scenarios that I have tested:

1) Webrick -> application thread
2) Apache/mod_ruby -> seperate ruby process
3) Lighttpd -> FCGI proxy (written in Ruby) -> seperate ruby process

Do you have any code you'd be willing to share?

I'm somewhat convinced that it's a problem with what I've written, so I
would not be surprised if others aren't having this problem, but, well,
I'm having this problem and I can't seem to figure out what the deal is.
I'm especially concerned that the behaviour is a bit weird -- just
hanging, instead of throwing an actual error -- as that makes it more
likely I'm doing something wrong.

Thanks,
Luke

 

[Luke Kanies]

However, I did want to pipe up and say that I have code that is regularly
being used to pass files over the wire as large as 100 megabytes, with no
problems. This is a web based application, and works just fine in three
different scenarios that I have tested:

1) Webrick -> application thread
2) Apache/mod_ruby -> seperate ruby process
3) Lighttpd -> FCGI proxy (written in Ruby) -> seperate ruby process

Because you don't mention XMLRPC here, I figure it might be of value to
point to the webrick/soap4r versions of my code:

http://reductivelabs.com/cgi-bin/puppet.cgi/file/language/trunk/lib/puppet/filebucket.rb?rev=331

http://reductivelabs.com/cgi-bin/puppet.cgi/file/language/trunk/test/bucket/tc_bucket.rb?rev=331

 

[NAKAMURA, Hiroshi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

This is an addendum to my earlier thread about web services and ruby,
wherein I was forced to accept the inevitability of using threads.

I've successfully got my client and server talking to each other for
small tasks, but I'm trying to expand to being able to pass (sometimes
large) files around and I've run into what seems almost like a
fundamental problem.

Basically, I'm trying to send a file over the wire, and for files above
about 20k in size I get an indefinite hang.

Hmm. I tried sending 2M< size binary file but it seems to work. Can I
see an example which causes indefinite hang?

I'm currently actually passing the file contents as an argument to the
method call; should I instead somehow attach the file somehow? Is there
an accepted way to pass files around in XML that I don't know about?

Do you see a sample of "SOAP with Attachment" at
http://dev.ctor.org/soap4r/browser/trunk/sample/soap/swa/ ?

Regards,
// NaHi
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.0 (Cygwin)

iD8DBQFC2QL2f6b33ts2dPkRAlRwAJ9Up8e/8oxWbSPnEz3EVZv9qRP0XwCfSTe/
7uVzIQhhnxLhIQagmjKzI3Q=
=IDmO
-----END PGP SIGNATURE-----