70 MB XML String ? XSL Transformation.

[CK]

Hello,
I have the 60 MB XML string and I am coding a program in Visual Basic
to run a XSL transformation on it. Currently, I'm using the Microsoft
standard MSXML 2.0 to create a DOM document, load the XML, and then
run the XSL transformation on it using the MSXML2.IXSLProcessor
interface.

The problem is that it kills my system because of an incredible amount
of memory that it requires to do this (around 850 MB). Are there any
other transformation methods that I can use in Visual Basic that will
take less memory then DOM?

Or perhaps some other method of transformation that uses SAX as input
rather than a DOM object ?

BTW, can't use .Net

ANy other ideas are appreciated...

 

[XxXxXxX]

What about spliting the XML instance into smaller chunks... run them
sequentially and then reassemble them.

 

[Andy Fish]

how dare you use the words 'visual basic' in this newsgroup. real XML
programmers always use java ;-))

AFAIK all existing XSLT processors use DOM (even the java ones). Because
XSLT allows you to jump all over the document, I think it would be very
difficut to produce one that didn't read the whole document in.

if your transformation is simple, I would suggest not using XSLT for it -
write a sax-based progam that processes it as a stream. (not sure if you can
do this with MSXML)

If you need to jump all over the document to get different parts, I think
you should probably re-atchitect the application to use a XML database or
lots of small pieces of XML held in flat files or RDBMS.

 

[Martin Boehm]

how dare you use the words 'visual basic' in this newsgroup.

<ignore-irony>What has the programming language of choice to do with
XML? You always have to use *one* language to approach the problem,

if your transformation is simple, I would suggest not using XSLT for
it - write a sax-based progam that processes it as a stream. (not
sure if you can do this with MSXML)

Of course you can.

If you need to jump all over the document to get different parts, I
think you should probably re-atchitect the application to use a XML
database or lots of small pieces of XML held in flat files or RDBMS.

Yes, sometimes XML ist _not_ the solution. *g*

Martin

 

[Julian F. Reschke]

how dare you use the words 'visual basic' in this newsgroup. real XML
programmers always use java ;-))

AFAIK all existing XSLT processors use DOM (even the java ones). Because
XSLT allows you to jump all over the document, I think it would be very
difficut to produce one that didn't read the whole document in.

Actually, must XSLT processors do NOT use DOM. What they use is an internal
representation of the XPath data model, most (*) of the time in RAM.

* one exception seems to be jd.xslt (http://www.aztecrider.com/xslt/).

...

Julian

 

[Andy Fish]

I stand corrected, thank you

I'm amazed that a 70mb file takes up 800mb of RAM - but then again just
opening it up in notepad would probably take 300mb

 

[Martin Boehm]

I'm amazed that a 70mb file takes up 800mb of RAM - but then again
just opening it up in notepad would probably take 300mb

If you build a tree of more or less complex objects from the plain text,
it is clear that it eats up more RAM than just reading it into a buffer.

Martin

 

[Richard Tobin]

I'm amazed that a 70mb file takes up 800mb of RAM - but then again
just opening it up in notepad would probably take 300mb

[/QUOTE]

If you build a tree of more or less complex objects from the plain text,
it is clear that it eats up more RAM than just reading it into a buffer.

Well, it doesn't have to be *that* much more. Element and attribute
names may be replaced with pointers to shared strings, and whitespace
and syntax characters will be discarded. A tree structure *could*
be smaller than the text.

On the other hand the characters will probably be represented as at
least 16 bits each, maybe 32, depending on the language and
implementation. There will be parent pointers, maybe position
information and so on.

-- Richard