Limited XML tidy

[Toby White]

I have a program which produces well-formed XML documents,
but takes several hours if not days to do so. It would
be useful to be able to take the incomplete output and
manipulate it as XML.

Clearly, however, the incomplete output will not be well-
formed, so before being able to manipulate it I need to
make it wellformed.

This is essentially fairly simple. I know it's well-formed
up to now - all I need to do is close unclosed tags. So,
I 've made a short function that will do this:

#!/usr/bin/python

import sys
import xml.sax

tagStack = []

closingTags = ""

class DodgyHandler(xml.sax.ContentHandler):
def startElement(self, tag, attributes):
tagStack.append(tag)

def endElement(self, tag):
tagStack.pop()


class DodgyErrorHandler(xml.sax.ErrorHandler):
def fatalError(self,exception):
global closingTags
tagStack.reverse()
for tag in tagStack:
closingTags += "</%s>" % tag
return closingTags


def finishXML(text):
p = xml.sax.make_parser()
p.setContentHandler(DodgyHandler())
p.setErrorHandler(DodgyErrorHandler())

for line in text:
p.feed(line)
p.close()

text.append(closingTags)


However - while this works for 90% of the cases I need,
it fails in the case where my incomplete output stops in the
middle of a tag (not to mention some other more arcane
places I don't really care about).

The problem is that when the sax handler raises an exception,
I can't see how to find out why. What I want to do is for
DodgyErrorHandler to do something different depending on
where we are in the course of parsing. Is there anyway
to get that information back from xml.sax (or indeed from
any other sax handler?)

Toby

 

[Magnus Lie Hetland]

[snip]

I do similar stuff in the new (upcoming) version of Atox
(atox.sf.net), which works with potentially ill-formed, partial XML
(in the form of PYX events) internally, and can take partial,
ill-formed XML as input.

The problem is that when the sax handler raises an exception,
I can't see how to find out why. What I want to do is for
DodgyErrorHandler to do something different depending on
where we are in the course of parsing. Is there anyway
to get that information back from xml.sax (or indeed from
any other sax handler?)

What I ended up doing was using an SGML parser (sgmlop) instead. It's
highly forgiving (even of illegal entities and the like) but gives me
the information I need. Might be worth a look in your app too?

 

[uche.ogbuji]

The problem is that when the sax handler raises an exception,
I can't see how to find out why. What I want to do is for
DodgyErrorHandler to do something different depending on
where we are in the course of parsing. Is there anyway
to get that information back from xml.sax (or indeed from
any other sax handler?)

You can get raw location information, yes. See:

http://www.xml.com/pub/a/2004/11/24/py-xml.html

But I don't think this is enough for you. You also need recovery,
which you're implementing in crude form.

I tend to agree with Magnus that using an SGML parser might be your
best bet. You might even be able to turn that SGML into XML using a
tool such as James Clark's SX:

http://www.jclark.com/sp/sx.htm

 

[Toby White]

I can't see how to find out why. What I want to do is for
DodgyErrorHandler to do something different depending on
where we are in the course of parsing. Is there anyway
to get that information back from xml.sax (or indeed from
any other sax handler?)

You can get raw location information, yes. See:

http://www.xml.com/pub/a/2004/11/24/py-xml.html

But I don't think this is enough for you. You also need recovery,
which you're implementing in crude form.

(If you're referring to the Locator objects), yes I'm aware
that's possible. But what I want is not my location in the
document, but for the parser to say "this is an error because
I am in the middle of a tag & the document ended", or "I
was in the middle of a text section and the document ended", or
"I was in the middle of an attribute value and the document
ended", etc, so that I can then construct a simple end to the
document, inserting quote marks, finishing the tag, and closing
all unclosed tags as appropriate.

I have just realised that I might be able to grab the message
that the exception gives me, look at the expat source code
and work out what parsing events cause which error messages.
Which is a bit round the houses, but I think ought to work.

I tend to agree with Magnus that using an SGML parser might be your
best bet. You might even be able to turn that SGML into XML using a
tool such as James Clark's SX:

http://www.jclark.com/sp/sx.htm

If I can't get my scheme above to work, I'll have a go. But I was
hoping to do this without requiring additional packages. And in
any case, it doesn't need to be perfectly robust. As long as it
handles 99% of cases, I'll be happy.