Iterparse and ElementTree confusion

[paul.sherwood]

Hi

Im trying to parse a large(150MB) xml file in order to extract specific
required records.

import sys
from elementtree.ElementTree import ElementTree

root = ElementTree(file=big.xml')

This works fine for smaller versions of the same xml file but...when i
attempted the above my PC goes to lala land, theres much HDD grinding
followed by "windows runnign low on virtual memory" popup after
10-15mins. Then just more grinding...for an hour before i gave up

XML file format:
<root>
<rubbish1>
.
.
</rubbish1>
.
.
<rubbishX>
.
.
</rubbishX>
<Products>
<Product ID="QU17861" UserTypeID="PH_QUOTE" QualifierID="Qualifier
root" ParentID="LIVE_AREA">
<Name QualifierID="Qualifier root">23172</Name>
<Description QualifierID="Qualifier root">Three Spot Rail Light
Brushed Chrome</Description>
<ClassificationReference ClassificationID="W2@Kitchen Lighting"
QualifierID="Qualifier root" Type="" />
<ProductReference ProductID="QU17749" QualifierID="Qualifier root"
Type="Accessory / Linked Product">
<Name QualifierID="Qualifier root">73520</Name>
.
.etc
</Product>
</Products>
</root>

Ok, i thought, surely theres a way to parse this thing in chucnks till
i get to the element i require then I'll reuse the ElementTree
goodness.

I found Iterparse

def parse_for_products(filename):

for event, elem in iterparse(filename):
if elem.tag == "Products":
root = ElementTree(elem)
print_all(root)
else:
elem.clear()

My problem is that if i pass the 'elem' found by iterparse then try to
print all attributes, children and tail text i only get
elem.tag....elem.keys returns nothing as do all of the other previously
useful elementtree methods.

Am i right in thinking that you can pass an element into ElementTree?
How might i manually iterate through <product>...</product> grabbing
everything?

 

[paul.sherwood]

Further to the above....

I pass my found element 'Product' to ElementTree's handy print all
function

def print_all(element):
root = element
#Create an iterator
iter = root.getiterator()
#Iterate
for element in iter:
#First the element tag name
print "Element:", element.tag
#Next the attributes (available on the instance itself using
#the Python dictionary protocol
if element.keys():
print "\tAttributes:"
for name, value in element.items():
print "\t\tName: '%s', Value: '%s'"%(name, value)
#Next the child elements and text
print "\tChildren:"
#Text that precedes all child elements (may be None)
if element.text:
text = element.text
text = len(text) > 40 and text[:40] + "..." or text
print "\t\tText:", repr(text)
if element.getchildren():
#Can also use: "for child in element.getchildren():"
for child in element:
#Child element tag name
print "\t\tElement", child.tag
#The "tail" on each child element consists of the text
#that comes after it in the parent element content, but
#before its next sibling.
if child.tail:
text = child.tail
text = len(text) > 40 and text[:40] + "..." or text
print "\t\tText:", repr(text)

if the element i pass to the above is from

root = ElementTree(file='somefile.xml')
iter = root.getiterator()
#Iterate
for element in iter:
if element.tag == 'Product':
print_all(element)

I print all of my required element's attributes, children, children's
children, etc

However if i pass the element i get from iterparse

for event, elem in iterparse(filename):
if elem.tag == "Products":
print_all(elem)
else:
elem.clear()

i only get the attributes for <product> and a list of its children, no
further iteration into the children and the children's children

what am i missing?

 

[Fredrik Lundh]

def parse_for_products(filename):

for event, elem in iterparse(filename):
if elem.tag == "Products":
root = ElementTree(elem)
print_all(root)
else:
elem.clear()

My problem is that if i pass the 'elem' found by iterparse then try to
print all attributes, children and tail text i only get
elem.tag....elem.keys returns nothing as do all of the other previously
useful elementtree methods.

Am i right in thinking that you can pass an element into ElementTree?
How might i manually iterate through <product>...</product> grabbing
everything?

by default, iterparse only returns "end" events, which means that the
iterator will visit the Products children before you see the Products
element itself. with the code above, this means that the children will
be nuked before you get around to process the parent.

depending on how much rubbish you have in the file, you can do

for event, elem in iterparse(filename):
if elem.tag == "Products":
process(elem)
elem.clear()

or

for event, elem in iterparse(filename):
if elem.tag == "Products":
process(elem)
elem.clear()
elif elem.tag in ("Rubbish1", "Rubbish2"):
elem.clear()

or

inside = False
for event, elem in iterparse(filename, events=("start", "end")):
if event == "start":
# we've seen the start tag for this element, but not
# necessarily the end tag
if elem.tag == "Products":
inside = True
else:
# we've seen the end tag
if elem.tag == "Products":
process(elem)
elem.clear()
inside = False
elif not inside:
elem.clear()

for more info, see

http://effbot.org/zone/element-iterparse.htm

</F>

 

[Steve M]

"when i attempted [to load 150MB xml file] my PC goes to lala land,
theres much HDD grinding followed by "windows runnign low on virtual
memory" popup after 10-15mins. Then just more grinding...for an hour
before i gave up"

I have had great success using SAX to parse large XML files. If you use
care your memory use will peak at a low number no matter how much XML
you chew through.

 

[paul.sherwood]

Ofcourse!

Thankyou so much. All is well, using the Iterparse takes a mere 3mins
to find the last product in the xml and this is probably due to my
checking 10000 products for a specific.

<looks at shoes>I feel rather honoured to have 'the' effbot reply to my
humble post....</looks at shoes>

Thanks again

Paul