XML::Simple in perl?

[Dan]

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

Should I consider using (learning)
Expat.c for increased performance?

How long would parsing a 250M XML
file take with Expat?

Thanks for any suggestions you can give,
Dan.

 

[Janek Schleicher]

yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

XML::Simple is not the only module on CPAN.
There are also
XML::Smart
XML:arser
XML::LibXML
....

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

Note that SAX is basical an API for XML triggering starting and ending
tags and similar. That's different to the previous mentioned modules, as
they also generate a huge tree view to the XML document.

But SAX can reduce the memory wastage and gain in time advantage.
In Perl you can use e.g. XML::SAX for it.

Should I consider using (learning)
Expat.c for increased performance?

XML:arser and XML::SAX::Expat are e.g. based on Expat, so you might use
the Perl interfaces if you want.



Greetings,
Janek

 

[Dan]

yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:




XML::Simple is not the only module on CPAN.
There are also
XML::Smart
XML:arser
XML::LibXML
...

Which ones have the same 'front end' as
XML simple? I would rather not change
code if I don't have to.

Which one uses least memory on big files?

Note that SAX is basical an API for XML triggering starting and ending
tags and similar. That's different to the previous mentioned modules, as
they also generate a huge tree view to the XML document.

But SAX can reduce the memory wastage and gain in time advantage.
In Perl you can use e.g. XML::SAX for it.




XML:arser and XML::SAX::Expat are e.g. based on Expat, so you might use
the Perl interfaces if you want.

I think I will end up doing this...

Is it any slower?

My requirements are extreemly simple, I just need
to print tab delimited lines to a file, so I was
thinking I could take this opportunity to try
to learn c....

Cheers,
Dan.

 

[Tad McClellan]

Which ones have the same 'front end' as
XML simple?

None of them.

The "simple" front end is why the module is called Simple.

I would rather not change
code if I don't have to.

Too late. :-(

Which one uses least memory on big files?

None of those ones.



That one.



There is also a mailing list specifically for doing XML processing
using Perl:

http://listserv.ActiveState.com/mailman/listinfo/perl-xml

 

[cp]

Using XML::Simple in perl is
extreemly slow to parse big
XML files (can be up to 250M,
taking ~1h).

XML::Simple is limited. The author points it out in the docs. It was
designed for a fairly specific purpose, parsing small configuration
files written in XML. It has since been expanded on, but it's core
remains, well, simple.

How can I increase my performance /
reduce my memory usage?

Is SAX the way forward?

The docs for XML::Simple Version 2.05 suggest that you can (as of
version 1.08) use a SAX parser with XML::Simple, the benefits of which
are:

Applications written to the SAX API can extract data
from huge XML documents without the memory overheads
of a DOM or tree API.

So you might read the docs, use a SAX parser, and see some speed
benefit. You might not.

 

[Dan]

I am now using XML:arser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing,
(i.e. I get a 1 instead of 19).

The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.

It is really confusing.

Also my Char event gets called 3 times
per tag, even though there are no new
lines anywhere in teh tag text.

Little frustrating problems....

Dan.

 

[Michel Rodriguez]

yDan wrote at Thu, 31 Jul 2003 12:41:24 +0100:
[XML::Simple is] extreemly slow to parse big [files]
There are also
XML::Smart
XML:arser
XML::LibXML
...

Which ones have the same 'front end' as XML simple?

None of them.

The "simple" front end is why the module is called Simple.

As a matter of fact XML::Smart has the same 'front end', and XML::Twig has a
method names 'simplify', which generates the same data structure as
XML::Simple, on a tree or on a sub-tree.

One possible cause for the problem might be that XML::Simple could be using
XML::SAX:urePerl as its parser, which is very slow. This depends on your
installation. If you have XML::LibXML or XML:arser installed you can set
the $XML::Simple:REFERRED_PARSER variable to tell it to use an other
parser, see the docs.

You might want to have a look at XML::Twig, which is specially designed for
big files (but I might be slightly biased ;--)

__
Michel Rodriguez
Perl & XML
http://xmltwig.com

 

[Tad McClellan]

I am now using XML:arser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing,

The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.

If you show us a short and complete program that we can run that
illustrates your problem, then we can surely help you solve
your problem.

It is really confusing.

Can't help with unseen code...

Also my Char event gets called 3 times
per tag,

That is "normal".

You'll get the PCDATA in dribs and drabs, so you need to keep collecting
it until you reach the end of the containing element.

even though there are no new
lines anywhere in teh tag text.

The concept of "lines" is not present in XML.

Remove every newline from an XML document, and it is still
an XML document.



[snip upside-down quoting]

 

[Dan]

Ta, the problem is fixed now,
I forgot to unset my global $currentTag
in the &endTag event handler, leading to
the 'dribs and drabs' below, which actually
belonged to outer tags (I was mistakenly
giving them to $currentTag).

With this bug gone I can now safely

$data{$currentTag} .= $text;

where I had been

$data{$currentTag} = $text if !$data{$currentTag};

Hence my occasional missing characters.

Thanks very much for all the kind help
and advice,

Regards,

Dan.

DIY GENOME...
perl -e '@A=qw(A T C G); for(1..10**6){print $A[rand(@A)]}' > \
myGenome.txt

I am now using XML:arser,
which is working nicely, apart
from the occasional weird behaviour,
in some cases characters go missing,

The error is persistant, i.e. not a random
caracter, but the same character each time
goes missing.

If you show us a short and complete program that we can run that
illustrates your problem, then we can surely help you solve
your problem.

It is really confusing.

Can't help with unseen code...

Also my Char event gets called 3 times
per tag,

That is "normal".

You'll get the PCDATA in dribs and drabs, so you need to keep collecting
it until you reach the end of the containing element.

even though there are no new
lines anywhere in teh tag text.

The concept of "lines" is not present in XML.

Remove every newline from an XML document, and it is still
an XML document.



[snip upside-down quoting]

 

[alskor]

caching for XML::Simple

I have another question about XML::Simple.
I'm trying to use its caching mechanism:

   my $xml = &XMLin($fileName, 
                KeyAttr => [], Cache => 'storable', 
                ForceArray => [$attributesList]);

it works like this: search for <file>.stor (cache) file. if not found, read the xml and store the cache file. if found, just read the cache file.
the problem is that I need to point to xml files located on read-only drive.
this server drive keeps xml files together with their ".stor" caches, but some of those xml files can lack their ".stor" caches yet. all the files on this server drive are read-only for clients.

is it possible to switch off "write" operation for this caching mechanism?
what I want for this caching on the client side is:
search for <file>.stor (cache) file. if not found, read the xml (DO NOT TRY TO store the cache file). if found, just read the cache file.