parsing multiple root element XML into text

[Percy Tambunan]

Hai, I would like to parse this multiple root element XML

<object class="EnumDnSched">
<field name="enumDn">
<value>343741014</value>
</field>
<field name="naptrFlags">
<value>nu</value>
</field>
</object>
<object class="EnumDnSched">
<field name="enumDn">
<value>343741015</value>
</field>
<field name="naptrFlags">
<value>nu</value>
</field>
</object>

into this
create enumdnsched 4.1.0.1.4.7.3.4.3.2.6.e164.arpa -set naptrFlags=nu
create enumdnsched 5.1.0.1.4.7.3.4.3.2.6.e164.arpa -set naptrFlags=nu

Anyone can great example how to do that, I would really appreciate it.

Thanks,
Percy

 

[Marko Rauhamaa]

Hai, I would like to parse this multiple root element XML

How about creating a file-like object that wraps the multi-root file
into a single-root document?


Marko

 

[Stefan Behnel]

Chris Angelico, 09.05.2014 11:02:

Easy fix might be to wrap it in <root> and </root>, which will give
you a new root.

ElementTree's XMLParser() can be use efficiently for this. Something like
this should work:

from xml.etree.ElementTree import XMLParser

parser = XMLParser()
parser.feed(b'<root>')
parser.feed(real_input_data)
parser.feed(b'</root>')
root = parser.close()

for subtree in root:
...

Stefan

 

[Chris Angelico]

Chris Angelico, 09.05.2014 11:02:

ElementTree's XMLParser() can be use efficiently for this. Something like
this should work:

from xml.etree.ElementTree import XMLParser

parser = XMLParser()
parser.feed(b'<root>')
parser.feed(real_input_data)
parser.feed(b'</root>')
root = parser.close()

for subtree in root:
...

That looks good to me

ChrisA

 

[Alain Ketterlin]

Hai, I would like to parse this multiple root element XML

<object class="EnumDnSched"> [...]
</object>
<object class="EnumDnSched"> [...]
</object>

Technically speaking, this is not a well-formed XML document (it is a
well-formed external general parsed entity, though). If you have other
XML processors in your workflow, they will/should reject it.

The easiest fix is to wrap this inside a root element (see other
messages in this thread), or use a DTD-declared entity to include this
fragment in a document.

-- Alain.

 

[Marko Rauhamaa]

Technically speaking, this is not a well-formed XML document (it is a
well-formed external general parsed entity, though). If you have other
XML processors in your workflow, they will/should reject it.

Sometimes the XML elements come through a pipe as an endless sequence.
You can still use the wrapping technique and a SAX parser. However, the
other option is to write a tiny XML scanner that identifies the end of
each element. Then, you can cut out the complete XML element and hand it
over to a DOM parser.

Such a scanner can be really small and nonrecursive because of the
welformedness rules of XML.


Marko

 

[Alain Ketterlin]

Sometimes the XML elements come through a pipe as an endless sequence.
You can still use the wrapping technique and a SAX parser. However, the
other option is to write a tiny XML scanner that identifies the end of
each element. Then, you can cut out the complete XML element and hand it
over to a DOM parser.

Well maybe, even though I see no point in doing so. If the whole
transaction is a single document and you need to get sub-elements on the
fly, just use the SAX parser: there is no need to use a "tiny XML
scanner" (whatever that is), and building a DOM for a part of the
document in your SAX handler is easy if needed (for the OP's case a
simple state machine would be enough, probably).

-- Alain.

 

[Marko Rauhamaa]

Well maybe, even though I see no point in doing so. If the whole
transaction is a single document and you need to get sub-elements on
the fly, just use the SAX parser: there is no need to use a "tiny XML
scanner" (whatever that is), and building a DOM for a part of the
document in your SAX handler is easy if needed (for the OP's case a
simple state machine would be enough, probably).

An example is <URL:
http://en.wikipedia.org/wiki/XMPP#XMPP_via_HTTP_and_WebSocket_transports>.

The "document" is potentially infinitely long. The elements are
messages.

The programmer would rather process the elements as DOM trees than
follow the meandering SAX parser.


Marko

 

[Marko Rauhamaa]

An example is <URL:
http://en.wikipedia.org/wiki/XMPP#XMPP_via_HTTP_and_WebSocket_transports>.

The "document" is potentially infinitely long. The elements are
messages.

The programmer would rather process the elements as DOM trees than
follow the meandering SAX parser.

In fact, the best thing would be if the DOM parser supported the use
case out of the box: give the partial, whole or oversize document to the
parser. If the document isn't complete, the parser should indicate the
need for more input. If there are bytes after the document is
successfully finished, the parser should leave the excess bytes in the
pipeline.

IOW, if the DOM parser knows full well where the document ends, why
must the application tell it to it?


Marko

 

[Stefan Behnel]

Marko Rauhamaa, 09.05.2014 14:38:

Marko Rauhamaa:

In fact, the best thing would be if the DOM parser supported the use
case out of the box: give the partial, whole or oversize document to the
parser. If the document isn't complete, the parser should indicate the
need for more input. If there are bytes after the document is
successfully finished, the parser should leave the excess bytes in the
pipeline.

IOW, if the DOM parser knows full well where the document ends, why
must the application tell it to it?

In fact, XMPP traffic has a root element. And I agree that a tree is much
easier to handle than SAX events. ElementTree has gained a nice API in
Py3.4 that supports this in a much saner way than SAX, using iterators.
Basically, you just dump in some data that you received and get back an
iterator over the elements (and their subtrees) that it generated from it.
Intercept on the right top elements and you get your next subtree as soon
as it's ready.

https://docs.python.org/3.4/library/xml.etree.elementtree.html#pull-api-for-non-blocking-parsing

It's also supported by recent versions of lxml, which additionally has easy
to use support for the sending side with its xmlfile() tool.

http://lxml.de/parsing.html#incremental-event-parsing

http://lxml.de/api.html#incremental-xml-generation

Stefan

 

[Marko Rauhamaa]

ElementTree has gained a nice API in Py3.4 that supports this in a
much saner way than SAX, using iterators.

Good to know.


Marko

 

[Alain Ketterlin]

An example is <URL:
http://en.wikipedia.org/wiki/XMPP#XMPP_via_HTTP_and_WebSocket_transports>.

The "document" is potentially infinitely long. The elements are
messages.

The programmer would rather process the elements as DOM trees than
follow the meandering SAX parser.

which does an exact traversal of potential the DOM tree... (assuming a
DOM is even defined on a non well-formed XML document).

Anyway, my point was only to warn the OP that he is not doing XML.

-- Alain.

 

[Marko Rauhamaa]

which does an exact traversal of potential the DOM tree... (assuming a
DOM is even defined on a non well-formed XML document).

Anyway, my point was only to warn the OP that he is not doing XML.

I consider that one of the multitude of flaws in XML.

Compare that with the close analogue: S expressions. Every lisp/scheme
command interface takes in a sequence of unframed S expressions and
interprets them in sequence. I have built similar interfaces using XML:
the command interface accepts a sequence of XML elements and acts on
them upon reaching the end of a complete element.


Marko

 

[Burak Arslan]

ElementTree has gained a nice API in
Py3.4 that supports this in a much saner way than SAX, using iterators.
Basically, you just dump in some data that you received and get back an
iterator over the elements (and their subtrees) that it generated from it.
Intercept on the right top elements and you get your next subtree as soon
as it's ready.

Hi Stefan,

Here's a small script:

events = etree.iterparse(istr, events=("start", "end"))
stack = deque()
for event, element in events:
if event == "start":
stack.append(element)
elif event == "end":
stack.pop()

if len(stack) == 0:
break

print(istr.tell(), "%5s, %4s, %s" % (event, element.tag, element.text))

where istr is an input-stream. (Fully working example:
https://gist.github.com/plq/025005a71e8135c46800)

I was expecting to have istr.tell() return the position where the first
root element ends, which would make it possible to continue parsing with
another call to etree.iterparse(). But istr.tell() returns the position
of EOF after the first call to next() on the iterator it returns.
Without the stack check, the loop eventually throws an exception and the
offset value in that exception is None.

So I'm lost here, how it'd possible to parse OP's document with lxml?

Best,
Burak

 

[Alain Ketterlin]

I consider that one of the multitude of flaws in XML.

I consider such use cases one of the multitude misuses of XML.

Compare that with the close analogue: S expressions. [...]

How do you specify the encoding of sexprs? How can you require that an
attribute value must match the value of an id-attribute? or whatever
insanely complex integrity rule that XML Schemas lets you express? And
so on.

If all you need to do is parse properly bracketed input, go with sexprs,
or json, or yaml, or pickle if both ends are python programs. Using XML
for such a trivial task is looking for trouble.

-- Alain.

 

[Marko Rauhamaa]

How do you specify the encoding of sexprs? How can you require that an
attribute value must match the value of an id-attribute? or whatever
insanely complex integrity rule that XML Schemas lets you express? And
so on.

I don't suppose there is a universal schema language available for S
expressions, nor do I suppose many people want such a thing. You specify
S expression encoding the way you specify Python library functions.

I think the worst part of XML is that you can't parse it without a DTD
or schema.

If all you need to do is parse properly bracketed input, go with
sexprs, or json, or yaml, or pickle if both ends are python programs.

I was very hopeful about json until I discovered they require the parser
to dynamically support five different character encodings.

XML at least standardized on UTF-8.

I have found ast.literal_eval() to be highly usable.

Pickle I would advise you to stay away from: <URL:
http://lwn.net/Articles/595352/>.


Marko

 

[Stefan Behnel]

Marko Rauhamaa, 09.05.2014 20:04:

I think the worst part of XML is that you can't parse it without a DTD
or schema.
Nonsense.


I was very hopeful about json until I discovered they require the parser
to dynamically support five different character encodings.

XML at least standardized on UTF-8.

Not really. Definitely not "standardized". It's just the default if there
is no explicit encoding or BOM. That doesn't keep you from passing around
XML in EBCDIC, if you feel like it.

I have found ast.literal_eval() to be highly usable.

Well, there's json.loads(), which is more commonly used for this task.

Stefan

 

[Stefan Behnel]

Burak Arslan, 09.05.2014 18:52:

Here's a small script:

A bit hard to read, though.

events = etree.iterparse(istr, events=("start", "end"))
stack = deque()
for event, element in events:
if event == "start":
stack.append(element)
elif event == "end":
stack.pop()

if len(stack) == 0:
break

print(istr.tell(), "%5s, %4s, %s" % (event, element.tag, element.text))

where istr is an input-stream. (Fully working example:
https://gist.github.com/plq/025005a71e8135c46800)

I was expecting to have istr.tell() return the position where the first
root element ends, which would make it possible to continue parsing with
another call to etree.iterparse(). But istr.tell() returns the position
of EOF after the first call to next() on the iterator it returns.

Correct, because it finished parsing. It controls the reading process and
reads ahead, that's how iterparse() works.

Without the stack check, the loop eventually throws an exception and the
offset value in that exception is None.

So I'm lost here, how it'd possible to parse OP's document with lxml?

See my earlier post. Instead of XMLParser, just use the XMLPullParser for
incremental (non-blocking) parsing and processing.

To make this clear, though: to use an XML parser, you need well formed XML,
and that means exactly one root element.

Stefan

 

[Percy Tambunan]

Easy fix might be to wrap it in <root> and </root>, which will give

you a new root. Would that help?



ChrisA

Thanks chris for the idea.
Any suggestion to make it print like this:

create enumdnsched 4.1.0.1.4.7.3.4.3.2.6.e164.arpa -set naptrFlags=nu
create enumdnsched 5.1.0.1.4.7.3.4.3.2.6.e164.arpa -set naptrFlags=nu

 

[Peter Otten]

Thanks chris for the idea.
Any suggestion to make it print like this:

create enumdnsched 4.1.0.1.4.7.3.4.3.2.6.e164.arpa -set naptrFlags=nu
create enumdnsched 5.1.0.1.4.7.3.4.3.2.6.e164.arpa -set naptrFlags=nu

[Stefan Behnel]

ElementTree's XMLParser() can be use efficiently for this. Something like
this should work:

from xml.etree.ElementTree import XMLParser

parser = XMLParser()
parser.feed(b'<root>')
parser.feed(real_input_data)
parser.feed(b'</root>')
root = parser.close()

for subtree in root:
...

Have you tried to integrate Stefan's example into your script? If so, what
is the current state of your code, and what problems prevented you from
completing it?

Expect help to fix these problems, but not necessarily a ready-to-run
solution.

If you have not made an attempt yet look here for ideas on how you can use
XPath to extract interesting data from the subtree:

https://docs.python.org/dev/library/xml.etree.elementtree.html#example