Need more info about problem resolving entity reference

David Karr · May 23, 2013

I have a Cygwin Perl script makes numerous REST api calls to a local service, parses the results from those, and makes other calls with that data. It also runs some of these calls in multiple threads, using LWP::UserAgent.

It mostly works, but I sometimes get errors like this:

-----------------------
caught error:
500 Can't connect to www.w3.org:80 (Operation now in progress) http://www.w3.org/TR/html4/strict.dtd
Handler couldn't resolve external entity at line 1, column 90, byte 92
error in processing external entity reference at line 1, column 90, byte 92:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
=========================================================================================^
<html>
<head>
at /usr/lib/perl5/vendor_perl/5.14/i686-cygwin-threads-64int/XML/Parser.pm line 187 thread 2

David Karr · May 24, 2013

This error comes from XML:arser. I assume you are invoking that

directly, to parse the REST response? What's happening is that

XML:arser sees a DOCTYPE declaration like

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"

"http://www.w3.org/TR/html4/strict.dtd">

and, like a good little SGML-derived XML parser, tries to fetch the DTD

(using LWP) so it can validate the rest of the file. For some reason,

when it tries to connect to www.w3.org to download the DTD file, the

connection is failing with EINPROGRESS. Since LWP isn't expecting that

error code, it throws an error.

So, what's the real problem? Well, first, that's an HTML doctype. You

can't, in general, parse HTML with an XML parser, so are you sure you're

getting the responses you expect? REST services are usually pretty good

about getting their Content-types right, so you ought to be able to

check for an XML Content-type before passing the data to XML:arser.

I'm completely certain that in these anomalous cases, I'm definitely not getting the response I expect. The problem with this error message is that it gives me absolutely no clue where in the script this is happening. I'm guessing that our back-end server gets confused in some cases, but it's hardto diagnose when I don't know what URL was being attempted, or where in the script it was done.

Second, you really don't want to keep fetching the DTDs like that. Does

the XML you're actually trying to parse use external DTDs? If not, then

you want to pass the NoLWP option to XML:arser, so that it doesn't

even try to fetch DTDs from the network. In the case of a public DTD

like HTML the attempt to load it as a local file will fail, of course,

but the parsing wasn't going to succeed anyway, because it wasn't XML.

That "NoLWP" option sounds useful, but it's somewhat moot here.

However, I'm slightly confused here, because the XML:arser

documentation seems to say it doesn't parse external DTDs by default.

It's possible I'm misunderstanding; I don't think I've used XML:arser

myself. Are you passing ParseParamEnt, and if so, why?

I don't know what "ParseParamEnt" is, so I imagine I'm not.

Third, you probably don't want to be using XML:arser at all. As you

can see, it's old and rather cronky, and while it's extremely solid code

it also takes a rather SGMLish approach to parsing XML. Most of the

time, with modern XML use, DTDs are not used, and instead the XML just

needs to be well-formed and properly namespaced. For this sort of thing

(small documents) I would use XML::LibXML (which, incidentally, also

includes a reasonable HTML parser); if a streaming model is more

appropriate, either because your documents may be ridiculously large or

simply because your program is structured that way, I would use one of

the SAX modules.

The funny thing about searching in CPAN is that there are no packages (I'm guessing) that say "do not use this, use something better". I'll take a look at XML::LibXML to see what it does for me.

Finally, fourth, I have no idea where that EINPROGRESS is coming from.

That error is supposed to be returned if a socket is connected while in

non-blocking mode, and the connection cannot be completed without

blocking; it's basically the equivalent of EAGAIN for connect(). This

means it shouldn't be possible to get that error without having asked

for it by setting nonblocking mode on the socket, which LWP does not

(normally) do.

Are you doing something peculiar which might cause this to happen?

Alternatively, it's possible this is some sort of Cygwin peculiarity,

which unfortunately may be difficult to track down; if you can isolate

the conditions where the error occurs it would be useful. (For instance,

does it tend to occur when the network goes down? When the network is

overloaded? When the DNS doesn't respond promptly?)

The script runs for perhaps 30-40 minutes, basically walking the entire data model of a REST api. It sends hundreds of requests to the (load-balanced) service, some from multiple threads. This kind of error happens several times during the run of the script, which means that the vast majority workwell enough. I ended up putting a hack into my "sendGet" sub that just checks for "DOCTYPE HTML" in the output and simply tries again, with a reasonable limit of retries. Almost all of the calls that detect this once or twice eventually get good data.

perl, xpath and dtd	1	Feb 20, 2008
PyXML, Sax, error in processing external entity reference	2	Feb 25, 2004
HTMLParser problem	4	Nov 16, 2004
Exception when calling a DOM method using a reference to it	15	Feb 14, 2004
Upgrading to CPAN.pm v1.76 install Bundle::CPAN fails	1	May 10, 2005
Sencha Touch--Support 2 browsers in just 228K!	64	Jul 16, 2010
RXParse module v.90 (by robic0)	0	May 29, 2006
RXParse module (by robic0), Version 0.1000	29	Apr 17, 2006

Need more info about problem resolving entity reference

David Karr

David Karr

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads