Webservices returning large amount of data - need suggestions

[DarthLuch]

I'd like to get suggestions on best practice way to return a large
amount of xml data using a webservice function.

Here's what I do today:
Using java, I write a function that will return an array of some
object. I write the function to loop through the database, getting each
record, perform any needed manipulations on it, then add each modified
record to an array. At the end, I marshall the array to a string (using
Castor) and return the xml-representation of it. I then use Apache Axis
to expose the function as a web-service. I know this is not a
well-though design, but it is easy to program. So now the problem I'm
having is that I run out of memory... With a large amount of data it
will eventually crash once it reaches its breaking point of adding so
many items to the array. I also know that taking the entire object and
marshalling it to a string to be returned is probably not the best way
either. Any suggestions and pointers would be appreciated.

My clients will usually deploy to a Tomcat server, using the default
settings. I'm not sure if there's even a way to increase the memory
allotted to it, but I'm not sure that is the best way to proceed anyway.

 

[seank76]

If I understand the problem correctly, you just need to be able to
return a huge string via webservice, right?
The first thing that comes to my mind is to implement iterator type of
service.
In other words, you would only return a predefined size of string and
the client will have to call "hasNext()" method to see if there are
more, etc.
This way you always know exactly how much memory is required regardless
of the length of the string.
I guess if the performance matters and multiple hits to the web-methods
are not desired, this might not be the ideal solution for you...

 

[Luch]

Yeah multiple hits to the webmethods is not desired. I also prefer that
the methods stay state(or session?) less, and each method is it's own
instance, etc.
Would like other ideas.