ElementTree - Why not part of the core?

[doug.bromley]

Why is the ElementTree API not a part of the Python core?
I've recently been developing a script for accessing the Miva API only
to find all the core API's provided by Python for parsing XML is messy
and complicated. Many of the examples I see for parsing the data using
these API's uses a similar additional Class for collapsing the XML data
into a more manageable format.
This is clearly not following the Python-way of clean, simple code and
easy development.

ElementTree on the other hand provides incredibly easy access to XML
elements and works in a more Pythonic way. Why has the API not been
included in the Python core?

 

[jelle]

Doug,

I agree with you, ElementTree is fast & pythonic.
Certainly does make sense to me.

 

[Steven Bethard]

ElementTree on the other hand provides incredibly easy access to XML
elements and works in a more Pythonic way. Why has the API not been
included in the Python core?

While I fully agree that ElementTree is far more Pythonic than the
dom-based stuff in the core, this issue has been discussed on
python-dev[1]. Fredrik Lundh's response:

shipping stable versions of ElementTree/cElementTree (or PIL, or
python-doc, or exemaker, or what else you might find useful) with
official Python releases is perfectly okay.

moving the main trunk and main development over to the Python CVS is
another thing, entirely.

I think some people were hoping that instead of adding these things to
the standard library, we would come up with a better package manager
that would make adding these things to your local library much simpler.

STeVe

[1]http://www.python.org/dev/summary/2005-06-01_2005-06-15.html#reorganising-the-standard-library-again

 

[Jarek Zgoda]

Steven Bethard napisa³(a):

ElementTree on the other hand provides incredibly easy access to XML
elements and works in a more Pythonic way. Why has the API not been
included in the Python core?

While I fully agree that ElementTree is far more Pythonic than the
dom-based stuff in the core, this issue has been discussed on
python-dev[1]. Fredrik Lundh's response:

shipping stable versions of ElementTree/cElementTree (or PIL, or
python-doc, or exemaker, or what else you might find useful) with
official Python releases is perfectly okay.

moving the main trunk and main development over to the Python CVS is
another thing, entirely.

I think some people were hoping that instead of adding these things to
the standard library, we would come up with a better package manager
that would make adding these things to your local library much simpler.

What about more intelligent installer, that after installing core
packages, would present user a list of recommended packages in few
groups (like "markup processing", "popular RDBMS interaction", "web
frameworks", "multimedia", etc.) and would be able to fetch and install
packages using PyPI descriptions? Unattended install with this feature
would be great.

 

[Fredrik Lundh]

ElementTree on the other hand provides incredibly easy access to XML
elements and works in a more Pythonic way. Why has the API not been
included in the Python core?

While I fully agree that ElementTree is far more Pythonic than the
dom-based stuff in the core, this issue has been discussed on
python-dev[1]. Fredrik Lundh's response:

shipping stable versions of ElementTree/cElementTree (or PIL, or
python-doc, or exemaker, or what else you might find useful) with
official Python releases is perfectly okay.

moving the main trunk and main development over to the Python CVS is
another thing, entirely.

I think some people were hoping that instead of adding these things to
the standard library, we would come up with a better package manager
that would make adding these things to your local library much simpler.

I still hope that the standard distribution will, in a not too distant future,
bundle more external libraries. as things are today, "including something
in the core" means that you have to transfer code and rights to the PSF.

as I've said many times, if the Linux folks can build distributions that con-
sists of thousands of individually maintained pieces, the Python distributors
should be able to handle a few dozen components.

</F>

 

[Simon Percivall]

Before that can happen we'll need some better management of co-existing
different versions of a package. You'll want to be able to use newer
versions of external packages without breakage in the standard library.

 

[rurpy]

Steven Bethard wrote:
--snip--

I think some people were hoping that instead of adding these things to
the standard library, we would come up with a better package manager
that would make adding these things to your local library much simpler.

STeVe

[1]http://www.python.org/dev/summary/2005-06-01_2005-06-15.html#reorganising-the-standard-library-again

A better package manager would be great but does not
replace having things in the core. Distributed code that
relies on external packages puts a significantly greater
burden on the user of the code. I suppose there are
ways to get around it but often it means at a minimum
the user will need to read the install instructions (which
seems to be asking a lot for some people.) And at some
companies, one has to jump though beauracratic hoops
for each external package installed. And I personally
stear away from packages that have a long list of
prerequisites.

With things I've written that I thought might be used
by others, I've gone to some lengths to use only modules
in the standard library.

 

[Mike Meyer]

as I've said many times, if the Linux folks can build distributions that con-
sists of thousands of individually maintained pieces, the Python distributors
should be able to handle a few dozen components.

Yes, but "distributers" is not necessarily "maintainers". How many
components are there in the Activestate Python distribution?
Enthoughts? Others?

The maintainers of Linux/gcc/etc. don't worry about creating a
distribution that includes lots of third party software. Why should
the maintainers of Python be any different?

<mike

 

[Alex Martelli]

seems to be asking a lot for some people.) And at some
companies, one has to jump though beauracratic hoops
for each external package installed. And I personally
stear away from packages that have a long list of
prerequisites.

Funny -- me, I prefer to steer away from companies with lots of
bureaucracy, instead!-)


Alex

 

[Magnus Lycka]

ElementTree on the other hand provides incredibly easy access to XML
elements and works in a more Pythonic way. Why has the API not been
included in the Python core?

I'd really like to see that too. Sure, it's fairly trivial to install
it, but each different package that needs to be installed from another
source, built and tested on multiple platforms etc, means more work.
For complex software systems, these kinds of configuration issues
take a lot of efforts.

We're deploying our software on a number of different platforms. We
certainly depend on Python, so a standard Python install will always
be included. Using standard library modules is for free. Using yet
another third party library has a cost, even if some Cheese Shop or
whatever might make the cost slightly lower.

We have very friendly configuration management guys here, so if I
insisted, I'm sure they'd let me have elementTree included, but I
hesitate to ask for it. It does add a burden on CM, increases the
risk that a deployment will fail, and from a code maintenance point
of view, it's a double edged sword: On one hand, it's a much better
API, but on the other hand, it's not documented in the standard
Python docs or in the Nutshell book, and a new developer would need
to figure out what this foreign beast was, and where to find docs.
(Alternatively, I'd have to (God forbid) document my actions.

 

[Fredrik Lundh]

We're deploying our software on a number of different platforms. We
certainly depend on Python, so a standard Python install will always
be included. Using standard library modules is for free. Using yet
another third party library has a cost, even if some Cheese Shop or
whatever might make the cost slightly lower.

We have very friendly configuration management guys here, so if I
insisted, I'm sure they'd let me have elementTree included, but I
hesitate to ask for it. It does add a burden on CM, increases the
risk that a deployment will fail, and from a code maintenance point
of view, it's a double edged sword: On one hand, it's a much better
API, but on the other hand, it's not documented in the standard
Python docs or in the Nutshell book, and a new developer would need
to figure out what this foreign beast was, and where to find docs.
(Alternatively, I'd have to (God forbid) document my actions.

on the other hand, you can bundle ElementTree with your applications simply
by including the ElementTree.py and (optionally) the ElementPath.py modules.

(you are allowed to add new Python modules to the project, I hope ;-)

(and yes, HTML documentation is included in the source distribution kit)


but seriously, given how easy it is to build things with distutils, I don't think your
configuration folks would have much trouble adding support for "anything that has
a setup file, and is reasonably self-contained" to their build scripts.

we usually use one large shell script with a bunch of "cd source; run setup install"
sections in it, e.g.

cd $PREFIX
cd src/griblib-2.1.1-20051125
$PYTHON setup.py install --prefix=$PREFIX

plus a Python script that does a sanity check at the end to make sure that the build
script actually managed to build and install everything we need. running the regression
test suite on the build is a good way to weed out build problems too; just make sure
you redirect the build output to a file, so you can check the logs afterwards.

all source kits are checked into the version management system, of course. no tarballs
are involved in the build process. just check things out and run the script, wait a couple
of minutes, and you're done.

</F>

 

[Fuzzyman]

ElementTree on the other hand provides incredibly easy access to XML
elements and works in a more Pythonic way. Why has the API not been
included in the Python core?

While I fully agree that ElementTree is far more Pythonic than the
dom-based stuff in the core, this issue has been discussed on
python-dev[1]. Fredrik Lundh's response:

shipping stable versions of ElementTree/cElementTree (or PIL, or
python-doc, or exemaker, or what else you might find useful) with
official Python releases is perfectly okay.

moving the main trunk and main development over to the Python CVS is
another thing, entirely.

I think some people were hoping that instead of adding these things to
the standard library, we would come up with a better package manager
that would make adding these things to your local library much simpler.

I still hope that the standard distribution will, in a not too distant future,
bundle more external libraries. as things are today, "including something
in the core" means that you have to transfer code and rights to the PSF.

as I've said many times, if the Linux folks can build distributions that con-
sists of thousands of individually maintained pieces, the Python distributors
should be able to handle a few dozen components.

I'd like to add my vote in favour of this.

There are a few popular extensions that most users would like easy
access to. PIL and ElementTree both fall into this category.

Thanks

Fuzzyman
http://www.voidspace.org.uk/python/index.shtml

 

[Giovanni Bajo]

I think some people were hoping that instead of adding these things
to
the standard library, we would come up with a better package manager
that would make adding these things to your local library much
simpler.

STeVe

[1]http://www.python.org/dev/summary/2005-06-01_2005-06-15.html#reorganising-th
e-standard-library-again

A better package manager would be great but does not
replace having things in the core. Distributed code that
relies on external packages puts a significantly greater
burden on the user of the code.

Seconded.

One thing I really fear about the otherwise great EasyInstall (and Python Eggs)
is that we could forget about


Let's not turn the Python standard library into the CPAN mess, where there are
5 different libraries for adding two numbers, so that it's then impossible to
grab a random perl program and read it, without going through 150 different man
pages you never saw before. I don't need 450 libraries to compute MD5, or to
zip a file, or 140 different implementations of random numbers. There will
always be external libraries for specific purposes, but I'd rather the standard
library to stay focused on provided a possibly restricted set of common
features with a decent interface/implementation.

This said, I'd also like to see ElementTree in the standard library. We already
have a SAX and a DOM, but we need a Pythonic XML library, and ElementTree is
just perfect.

 

[Giovanni Bajo]

One thing I really fear about the otherwise great EasyInstall (and
Python Eggs) is that we could forget about...

.... how important is to have a standard library. The fact that it's easy to
install external modules shouldn't make us drop the standard library. A
standard library means a great uniformity across programs. Whenever I open a
Python program which uses ZipFile, or socket, or re, I can read it
*immediately*. If it uses an external library / framework, I have to go study
its manual and documentation. Proliferation of external libraries is good, but
people should normally use the standard library modules for uniformity.

In other words, I disagree with this message:
http://mail.python.org/pipermail/python-dev/2005-June/054102.html

My personal hope is that Python 3.0 will have a nice cleaned-up standard
library with even more features than Python 2.x. As I said the in other
message, let's not end up into the CPAN mess just because it's now technically
easier!

 

[Magnus Lycka]

but seriously, given how easy it is to build things with distutils, I don't think your
configuration folks would have much trouble adding support for "anything that has
a setup file, and is reasonably self-contained" to their build scripts.

True. It's one more thing to keep track of though. It's a separate
configuration item. We need to keep track of which version to use,
and it needs to get built, installed and validated on all supported
platforms.

I already include Twisted (and thus ZopeInterface) like that. We
sometimes need high speed in XML parsing, so I'd like to use
cElementTree, but just like the two other packages, I assume that
it's unproblematic to build the included C code. So, you're right,
it's not much trouble, just two more pebbles (cElementTree and
ElementTree) on a fairly heavy load, where stuff like Oracle and MQ
Series are more like boulders.

Right now, I would like to improve the performace of a program that
does a lot of XML parsing. I haven't profiled that yet, so I don't
know if XML parsing is the bottle neck, but if I can improve perfor-
mance significantly with cElementTree, I'll certainly push for that.


/Magnus

 

[skip]

Magnus> I'd really like to see that too. Sure, it's fairly trivial to
Magnus> install it, but each different package that needs to be
Magnus> installed from another source, built and tested on multiple
Magnus> platforms etc, means more work.

More work than reimplementing the functionality or living with a miserable
API (I'm thinking normal DOM APIs)?

Skip

 

[skip]

I think the key here is ElementTree's Pythoninc API. While it's clearly
possible to install it as a third-party package, I think there's a clear
best-of-breed aspect here that suggests it belongs in the standard
distribution simply to discourage continued use of DOM-based APIs.

Skip

 

[Jeremy Hylton]

I still hope that the standard distribution will, in a not too distant future,

bundle more external libraries. as things are today, "including something
in the core" means that you have to transfer code and rights to the PSF.

Your description of how to include something in the core isn't entirely
accurate. If you visit http://www.python.org/psf/contrib.html, you'll
see that you don't need to transfer your rights to the PSF. Rather,
you need to sign an agreement giving the PSF the rights to relicense
the code. The PSF currently require that the original submission be
licensed under one of two open source licenses. Personally, I can
imagine accepting a wider range of initial licenses in the future.

As for transferring the code, there needs to be a copy in the Python
source control system, sure. There's some unavoidable complexity
involved in managing a software distribution composed of third party
software packages. At the very least, you've got the original sources
and the copy in the distribution package, which leads to a
synchronization problem.

Jeremy

 

[Steven Bethard]

I think the key here is ElementTree's Pythoninc API. While it's clearly
possible to install it as a third-party package, I think there's a clear
best-of-breed aspect here that suggests it belongs in the standard
distribution simply to discourage continued use of DOM-based APIs.

I second this. Guido has said many times that the stdlib is for
best-of-breed modules that have proven themselves in the wild.
ElementTree has proven itself in the wild and is clearly best-of-breed.
And dramatically better (IMHO) than the APIs currently included in the
stdlib[1].

I don't have a whole lot of free time, and I'm not sure exactly how I
could help, but if there's anything I could do that would help get
ElementTree into the stdlib, let me know.

STeVe

[1] If I had my way, we'd deprecate and then remove the current Python
xml modules. But of course then people would complain that Python
doesn't have a SAX or DOM API. Of course we could tell them that they
don't need it and that ElementTree is easier, but I'm not sure people
really want to fight that battle.

 

[Magnus Lycka]

Magnus> I'd really like to see that too. Sure, it's fairly trivial to
Magnus> install it, but each different package that needs to be
Magnus> installed from another source, built and tested on multiple
Magnus> platforms etc, means more work.

More work than reimplementing the functionality or living with a miserable
API (I'm thinking normal DOM APIs)?

More work for the CM people. Less work for me.