Which HTMLParser?

[Tuang]

The library docs show that there is an HTMLParser module and an
htmllib module, both of which apparently contain classes named
"HTMLParser". There is a bit of decription of differences, but it
still doesn't seem clear to me what the intent is.

Which one is the best choice for parsing arbitrary real-life Web
pages? I get the feeling that maybe the HTMLParser module is the more
recent, more practical utility, while the htmllib version is the older
one, retained for backward compatibility, but I'm not sure. The docs
don't exactly say that.

Any recommendations or clarifications of what's going on would be
helpful.

Thanks.

 

[Jarek Zgoda]

Which one is the best choice for parsing arbitrary real-life Web
pages? I get the feeling that maybe the HTMLParser module is the more
recent, more practical utility, while the htmllib version is the older
one, retained for backward compatibility, but I'm not sure. The docs
don't exactly say that.

Any recommendations or clarifications of what's going on would be
helpful.

If you are not sure that your source is valid HTML, use SGML parser
instead. Personally I recommend F. Lundh's sgmlop -- fast, robust and
well-written piece of software, real Meisterstueck. Works perfectly on
Unix, Windows and IBM iSeries (formerly AS/400).

 

[Rene Pijlman]

Tuang:

The library docs show that there is an HTMLParser module and an
htmllib module, both of which apparently contain classes named
"HTMLParser". There is a bit of decription of differences, but it
still doesn't seem clear to me what the intent is.

I think the intent is to use HTMLParser. Its newer, and its documentation
doesn't scare you off with phrases like "HTML 2.0" and "SGML"

Which one is the best choice for parsing arbitrary real-life Web pages?

Neither! Real-life web pages are typically not HTML-parseable. Try tyding
it up a bit first. See http://groups.google.nl/groups?th=58cd394d2e71137f

 

[John J. Lee]

If you are not sure that your source is valid HTML, use SGML parser
instead.

Note that htmllib is a simple subclass of sgmllib, so the results you
get from sgmllib will be the same as for htmllib as far as this
concern goes.

HTMLParser.HTMLParser can cope better with XHTML, and treats optional
or missing start/end tags more simply (ie. better) than sgmllib /
htmllib.

Personally I recommend F. Lundh's sgmlop -- fast, robust and
well-written piece of software, real Meisterstueck. Works perfectly on
Unix, Windows and IBM iSeries (formerly AS/400).

I don't think it's any more lenient, though. And harder to modify.

Use mxTidy or uTidylib to clean bad HTML.


John