Is there a HTML parser who can reconstruct the original html EXACTLY?

ioscas · Jan 23, 2008

Hi, I am looking for a HTML parser who can parse a given page into
a DOM tree, and can reconstruct the exact original html sources.
Strictly speaking, I should be allowed to retrieve the original
sources at each internal nodes of the DOM tree.
I have tried Beautiful Soup who is really nice when dealing with
those god damned ill-formed documents, but it's a pity for me to find
that this guy cannot retrieve original sources due to its great tidy
job.
Since Beautiful Soup, like most of the other HTML parsers in
python, is a subclass of sgmllib.SGMLParser to some extent, I have
investigated the source code of sgmllib.SGMLParser, see if there is
anything I can do to tell Beautiful Soup where he can find every tag
segment from HTML source, but this will be a time-consuming job.
so... any ideas?

cheers
kai liu

A.T.Hofkamp · Jan 23, 2008

Hi, I am looking for a HTML parser who can parse a given page into
a DOM tree, and can reconstruct the exact original html sources.

Why not keep a copy of the original data instead?

That would be VERY MUCH SIMPLER than trying to reconstruct a parsed tree back
to original source text.

sincerely,
Albert

kliu · Jan 23, 2008

Why not keep a copy of the original data instead?

That would be VERY MUCH SIMPLER than trying to reconstruct a parsed tree back
to original source text.

sincerely,
Albert

Thank u for your reply. but what I really need is the mapping between
each DOM nodes and
the corresponding original source segment.

Paul Boddie · Jan 23, 2008

Thank u for your reply. but what I really need is the mapping between
each DOM nodes and the corresponding original source segment.

At the risk of promoting unfashionable DOM technologies, you can at
least serialise fragments of the DOM in libxml2dom [1]:

import libxml2dom
d = libxml2dom.parseURI("http://www.diveintopython.org/", html=1)
print d.xpath("//p")[7].toString()

Storage and retrieval of the original line and offset information may
be supported by libxml2, but such information isn't exposed by
libxml2dom.

Paul

[1] http://www.python.org/pypi/libxml2dom

Stefan Behnel · Jan 23, 2008

Hi,

what I really need is the mapping between each DOM nodes and
the corresponding original source segment.

I don't think that will be easy to achieve. You could get away with a parser
that provides access to the position of an element in the source, and then map
changes back into the document. But that won't work well in the case where the
parser inserts or deletes content to fix up the structure.

Anyway, the normal focus of broken HTML parsing is in fixing the source
document, not in writing out a broken document. Maybe we could help you better
if you explained what your actual intention is?

Stefan

A.T.Hofkamp · Jan 23, 2008

Thank u for your reply. but what I really need is the mapping between
each DOM nodes and
the corresponding original source segment.

Why do you think there is a simple one-to-one relation between nodes in some
abstract DOM tree, and pieces of source?, For example, the outermost tag
<HTML>...</HTML> is not an explicit point in the tree. If if it is, what piece
of source should be attached to it? Everything? Just the text before and after
it? If so, what about the source text of the second tag? Last but not least,
what do you intend to do with the source-text before the <HTML> and after
the </HTML> tags?

In other words, you are going to have a huge problem deciding what
"corresponding original source segment" means for each tag. This is exactly the
reason why current tools do not do what you want.

If you really want this, you probably have to do it yourself mostly from
scratch (ie starting with a parsing framework and writing a custom parser
yourself). That usually boils down to attaching source text to tokens in the
lexical parsing phase. If you have a good understanding of the meaning of
"corresponding original source segment", AND you have perfect HTML, this is
doable, but not very nice.

There exist parsers that can do what you want IF YOU HAVE PERFECT HTML, but
using those tools implies a very steep learning curve of about 2-3 months under
the assumption that you know functional languages (if you don't, add 2-3 months
or so steep learning curve

).

If you don't have perfect HTML, you are probably more or less lost. Most tools
cannot deal with that situation, and those that can do smart re-shuffling to
make things parsable, which means there is really no one-to-one mapping any
more (after re-shuffling).

In other words, I think you really don't want what you want, at least not in
the way that you consider now.

Please give us information about your goal, so we can think about alternative
approaches to solve your problem.

sincerely,
Albert

Fuzzyman · Jan 23, 2008

[email protected] said:
Hi, I am looking for a HTML parser who can parse a given page into
a DOM tree, and can reconstruct the exact original html sources.
Strictly speaking, I should be allowed to retrieve the original
sources at each internal nodes of the DOM tree.
I have tried Beautiful Soup who is really nice when dealing with
those god damned ill-formed documents, but it's a pity for me to find
that this guy cannot retrieve original sources due to its great tidy
job.
Since Beautiful Soup, like most of the other HTML parsers in
python, is a subclass of sgmllib.SGMLParser to some extent, I have
investigated the source code of sgmllib.SGMLParser, see if there is
anything I can do to tell Beautiful Soup where he can find every tag
segment from HTML source, but this will be a time-consuming job.
so... any ideas?

A while ago I had a similar need, but my solution may not solve your
problem.

I wanted to rewrite URLs contained in links and images etc, but not
modify any of the rest of the HTML. I created an HTML parser (based on
sgmllib) with callbacks as it encounters tags and attributes etc.

It is easy to process a stream without 'damaging' the beautiful
orginal structure of crap HTML - but it doesn't provide a DOM.

http://www.voidspace.org.uk/python/recipebook.shtml#scraper

All the best,

Michael Foord
http://www.manning.com/foord

[Newbie] Is there any method to urlretrieve to a file the html source	3	Feb 25, 2008
Who owns the patent for HTML, if there is one?	27	Mar 30, 2007
How to parser the html dtd?	3	Oct 17, 2006
[ANN] Hpricot 0.6 -- the swift, delightful HTML parser	0	Jun 16, 2007
HTML Correctness and Validators	7	Dec 29, 2008
DOM with HTML	1	Jul 1, 2003
What is the best html to latex program on the market or the internet ?	11	Oct 22, 2007
This seems to crash my program and gives me errors on the #include statements	5	Sep 4, 2006

Is there a HTML parser who can reconstruct the original html EXACTLY?

ioscas

A.T.Hofkamp

kliu

Paul Boddie

Stefan Behnel

A.T.Hofkamp

Fuzzyman

Ask a Question

Similar Threads

Members online

Forum statistics

Latest Threads