Webservices and large chunks of binary data

[lord.zoltar]

We have a webservice that currently takes in a small chunk of binary
data (usually < 1MB) as a part of the message. There is a good chance
we will need to take much larger chunks of binary data (up to 20 MB).
I'm not a web services pro so I'm wondering if anyone with more
experience can say if this is a good idea, or bad idea; and if a bad
idea, what alternate solutions are there? We are stuck with using a
webservice for this application (I would think FTP would be better for
large file transfers but it doesn't look like we'll be able to do
that), and any changes to the web service will require changes to the
client application that sends us the messages (it's maintained by a
third party, but they'd probably be OK with a change to the wsdl if it
isn't TOO drastic). Any ideas?
TIA!

 

[Qu0ll]

We have a webservice that currently takes in a small chunk of binary
data (usually < 1MB) as a part of the message. There is a good chance
we will need to take much larger chunks of binary data (up to 20 MB).
I'm not a web services pro so I'm wondering if anyone with more
experience can say if this is a good idea, or bad idea; and if a bad
idea, what alternate solutions are there? We are stuck with using a
webservice for this application (I would think FTP would be better for
large file transfers but it doesn't look like we'll be able to do
that), and any changes to the web service will require changes to the
client application that sends us the messages (it's maintained by a
third party, but they'd probably be OK with a change to the wsdl if it
isn't TOO drastic). Any ideas?
TIA!

I think you might be better off using SAAJ and multiple SOAP messages, each
with a "chunk" of the data of a manageable size (e.g. 512K or similar) and
then reconstitute the entire data block programmatically after having
received all the chunks. That way you are able to provide a progress bar or
similar feedback and it also means that it's easier to abort the
transmission after a certain amount of data has been transmitted without
losing what's already come through.

--
And loving it,

-Qu0ll (Rare, not extinct)
_________________________________________________
(e-mail address removed)
[Replace the "SixFour" with numbers to email me]

 

[Owen Jacobson]

We have a webservice that currently takes in a small chunk of binary
data (usually < 1MB) as a part of the message.

Do you mean a SOAP endpoint, or do you mean web service in a more
abstract sense?

There is a good chance
we will need to take much larger chunks of binary data (up to 20 MB).
I'm not a web services pro so I'm wondering if anyone with more
experience can say if this is a good idea, or bad idea; and if a bad
idea, what alternate solutions are there?

Don't use SOAP messages for this. HTTP has reasonably good support for
moving binary data around, either as the request body or as a part of
the request body (see for example the way forms with file inputs are
handled).

We are stuck with using a
webservice for this application (I would think FTP would be better for
large file transfers but it doesn't look like we'll be able to do
that), and any changes to the web service will require changes to the
client application that sends us the messages (it's maintained by a
third party, but they'd probably be OK with a change to the wsdl if it
isn't TOO drastic). Any ideas?

WSDL? Ah, SOAP then. You poor bastard.

I'm not too up on what the SOAP HTTP binding allows and doesn't allow,
but you may be able to mix SOAP and a binary attachment using a MIME
multipart request body. Maybe. Alternately, you could *not* use SOAP
for the moving of files, which is probably a better solution from
performance and code clarity points of view.

This is one of the places where a simpler variety of web service
actually works well. I've designed a few services such that each
logical action or view has a distinct URL with clearly-defined
semantics for HTTP operations. A simple forum service might have
http://www.example.com/forums/thread/1056/reply as a POSTable URI
accepting a simple XML message as the POST body, and
http://www.example.com/forums/profile/my/upload-avatar as a POSTable
URI accepting an image file as-is to be used as the current user's
avatar.

The biggest problem with this format of web service is that there are
few tools for generating client stubs for it. Very few WSDL processors
handle anything other than SOAP-HTTP and maybe SOAP-SMTP. This puts a
much larger burden on clear, concise written API documentation. In the
reality I occupy, the generated stubs for SOAP interfaces have never
been as compatible as tool vendors would like to claim they are, so
it's not much of a difference for me, but your reality may be different.

-o

 

[Qu0ll]

Do you mean a SOAP endpoint, or do you mean web service in a more abstract
sense?


Don't use SOAP messages for this. HTTP has reasonably good support for
moving binary data around, either as the request body or as a part of the
request body (see for example the way forms with file inputs are handled).


WSDL? Ah, SOAP then. You poor bastard.

I'm not too up on what the SOAP HTTP binding allows and doesn't allow, but
you may be able to mix SOAP and a binary attachment using a MIME multipart
request body. Maybe. Alternately, you could *not* use SOAP for the
moving of files, which is probably a better solution from performance and
code clarity points of view.

Just use SAAJ (SOAP with Attachments API for Java) with the binary data as
an attachment.

--
And loving it,

-Qu0ll (Rare, not extinct)
_________________________________________________
(e-mail address removed)
[Replace the "SixFour" with numbers to email me]