Large Size XML files for data transfer

[RAJ]

Hi,
Need some advice with regards to XML files. Here is what I am trying to
achieve:

I have an SQL Server 7.0 database and a Web service running on a Windows
2003 enterprise machine (for development). The web service (developed using
VB.NET) creates an XML file which contains most of the database table. The
creation of this file takes less then couple of minutes. This XML is about
57 MB in size and is used to build a local database. The file is then zipped
to about 2.5 MB and sent to the calling application.

On the client the file is unzipped to 57 MB file, before the application
starts reading it. I am using the load method of the DOM object model to
check whether it is a well formed XML. The problem starts as soon as I try
to load this XML file. The client machine becomes unresponsive for a long
time and the whole process from there on is very slow as I assume that the
XML is loaded into memory while it is parsed. I have to read the whole XML
file and do not need to search for any nodes/tags inside the XML.

I would like to know whether it is right to use XML in this scenario? Am I
better off by generating a simple delimited file instead of an XML file? The
size of the XML file is at least 3 folds as compared to any other delimited
file.

Any advice in this regard will be helpful.

Thanks and regards,

RAJ

 

[bruce barker]

you should use an xmlreader to parse the file. this will only require a
small amout of the xml (read buufer) to be in memory at a time.

-- bruce (sqlwork.com)

 

[RAJ]

Thanks a lot. Your help is much appreciated.

regards,
RAJ