Understanding .pth in site-packages

[Josh English]

(This may be a shortened double post)

I have a development version of a library in c:\dev\XmlDB\xmldb

After testing the setup script I also have c:\python27\lib\site-packages\xmldb

Now I'm continuing to develop it and simultaneously building an application with it.

I thought I could plug into my site-packages directory a file called xmldb.pth with:

c:\dev\XmlDB\xmldb

which should redirect import statements to the development version of the library.

This doesn't seem to work.

Is there a better way to redirect import statements without messing with the system path or the PYTHONPATH variable?

Josh

 

[Philip Semanchuk]

(This may be a shortened double post)

I have a development version of a library in c:\dev\XmlDB\xmldb

After testing the setup script I also have c:\python27\lib\site-packages\xmldb

Now I'm continuing to develop it and simultaneously building an application with it.

I thought I could plug into my site-packages directory a file called xmldb.pth with:

c:\dev\XmlDB\xmldb

which should redirect import statements to the development version of the library.

This doesn't seem to work.

xmldb.pth should contain the directory that contains xmldb:
c:\dev\XmlDB

Examining sys.path at runtime probably would have helped you to debug the effect of your .pth file.

On another note, I don't know if the behavior of 'import xmldb' is defined when xmldb is present both as a directory in site-pacakges and also as a .pth file. You're essentially giving Python two choices from where to import xmldb, and I don't know which Python will choose. It may be arbitrary. I've looked for some sort of statement on this topic in the documentation, but haven't come across it yet.

Is there a better way to redirect import statements without messing with the system path or the PYTHONPATH variable?

Personally I have never used PYTHONPATH.


Hope this helps
Philip

 

[Peter Otten]

I have a development version of a library in c:\dev\XmlDB\xmldb

After testing the setup script I also have
c:\python27\lib\site-packages\xmldb

Now I'm continuing to develop it and simultaneously building an
application with it.

I thought I could plug into my site-packages directory a file called
xmldb.pth with:

c:\dev\XmlDB\xmldb

which should redirect import statements to the development version of the
library.

This doesn't seem to work.

You have to put the directory containing the package into the pth-file.
That's probably

c:\dev\XmlDB

in your case. Also, Python will stop at the first matching module or
package; if you keep c:\python27\lib\site-packages\xmldb that will shadow
c:\dev\XmlDB\xmldb.

%APPDATA%/Python/Python26/site-packages

may be a good place for the pth-file (I'm not on Windows and too lazy to
figure out where %APPDATA% actually is. The PEP
http://www.python.org/dev/peps/pep-0370/ may help)

 

[Josh English]

Philip,

Yes, the proper path should be c:\dev\XmlDB, which has the setup.py, xmldb subfolder, the docs subfolder, and example subfolder, and the other text files proscribed by the package development folder.

I could only get it to work, though, by renaming the xmldb folder in the site-packages directory, and deleting the egg file created in the site-packages directory.

Why the egg file, which doesn't list any paths, would interfere I do not know.

But with those changes, the xmldb.pth file is being read.

So I think the preferred search order is:

1. a folder in the site-packages directory
2. an Egg file (still unsure why)
3. A .pth file

It's a strange juju that I haven't figured out yet.

Thanks for the hint.

Josh

 

[Josh English]

Philip,

Yes, the proper path should be c:\dev\XmlDB, which has the setup.py, xmldb subfolder, the docs subfolder, and example subfolder, and the other text files proscribed by the package development folder.

I could only get it to work, though, by renaming the xmldb folder in the site-packages directory, and deleting the egg file created in the site-packages directory.

Why the egg file, which doesn't list any paths, would interfere I do not know.

But with those changes, the xmldb.pth file is being read.

So I think the preferred search order is:

1. a folder in the site-packages directory
2. an Egg file (still unsure why)
3. A .pth file

It's a strange juju that I haven't figured out yet.

Thanks for the hint.

Josh

 

[Philip Semanchuk]

Philip,

Yes, the proper path should be c:\dev\XmlDB, which has the setup.py, xmldb subfolder, the docs subfolder, and example subfolder, and the other text files proscribed by the package development folder.

I could only get it to work, though, by renaming the xmldb folder in the site-packages directory, and deleting the egg file created in the site-packages directory.

Why the egg file, which doesn't list any paths, would interfere I do not know.

But with those changes, the xmldb.pth file is being read.

So I think the preferred search order is:

1. a folder in the site-packages directory
2. an Egg file (still unsure why)
3. A .pth file

That might be implementation-dependent or it might even come down to something as simple as the in which order the operating system returns files/directories when asked for a listing. In other words, unless you can find something in the documentation (or Python's import implementation) that confirms your preferred search order observation, I would not count on it working the same way with all systems, all Pythons, or even all directory names.




Good luck
Philip

 

[OKB (not okblacke)]

Philip,

Yes, the proper path should be c:\dev\XmlDB, which has the
setup.py, xmldb subfolder, the docs subfolder, and example
subfolder, and the other text files proscribed by the package
development folder.

I could only get it to work, though, by renaming the xmldb folder
in the site-packages directory, and deleting the egg file created
in the site-packages directory.

Why the egg file, which doesn't list any paths, would interfere I
do not know.

But with those changes, the xmldb.pth file is being read.

So I think the preferred search order is:

1. a folder in the site-packages directory
2. an Egg file (still unsure why)
3. A .pth file

You say that the egg file was created by the setup script for the
library. Are you sure that this script did not also create or modify a
..pth file of its own, adding the egg to the path?

.pth files do not "redirect" imports from site-packages; they add
EXTRA directories to sys.path. Also note that this means the .pth file
itself is not part of the search path; it's not like you shadow a
package xyz by creating a .pth file xyz.pth "instead". A single .pth
file can list multiple directories, and it's those directories that are
added to the path.

I'm not sure how your package is set up, but easy_install, for
instance, creates an easy_install.pth file in site-packages. This file
contains references to egg files (or, at least in my case, .egg
directories created by unpacking the eggs) for each package installed
with easy_install. As far as I'm aware, Python doesn't have special
rules for putting egg files in the search path, so my guess is that
it's something like that: the "setup script" is creating a .pth file (or
modifying an existing .pth file) to add the egg to the path.

Read http://docs.python.org/library/site.html for the description
of how .PTH files work. I don't think there is a general way to
globally "shadow" a package that exists in site-packages. However,
according to the docs the .pth files are added in alphabetical order, so
if it is indeed easy_install.pth that is adding your egg, you could hack
around it by making a file with an alphabetically earlier name (e.g.,
a_xmldb.pth).

--
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is
no path, and leave a trail."
--author unknown

 

[Terry Reedy]

That might be implementation-dependent or it might even come down to
something as simple as the in which order the operating system
returns files/directories when asked for a listing.

Doc says first match, and I presume that includes first match within a
directory.

 

[Philip Semanchuk]

Doc says first match, and I presume that includes first match within a directory.

First match using which ordering? Do the docs clarify that?


Thanks
Philip

 

[Dennis Lee Bieber]

So I think the preferred search order is:

1. a folder in the site-packages directory
2. an Egg file (still unsure why)
3. A .pth file

It's a strange juju that I haven't figured out yet.

.egg files, as I recall, are basically just ZIP compressed files --
and since Python can (again, as I recall) import from a ZIP compressed
file, .egg files get added to sys.path

(just open an interactive Python session and do

import sys
sys.path

and review the results)

 

[Dennis Lee Bieber]

%APPDATA%/Python/Python26/site-packages

may be a good place for the pth-file (I'm not on Windows and too lazy to
figure out where %APPDATA% actually is. The PEP
http://www.python.org/dev/peps/pep-0370/ may help)

%APPDATA% is a user specific directory, on WinXP:
C:\Documents and Settings\<username>\Application Data

If PEP-370 has been implemented, I'm running too old of a Python for
it <G>; my site.py doesn't have the control logic for it.

 

[Josh English]

OKB,

The setup.py script created the egg, but not the .pth file. I created that myself.

Thank you for clarifying about how .pth works. I know "redirect imports" was the wrong phrase, but it worked in my head at the time. It appears, at least on my system, that Python will find site-packages/foo before it finds and reads site-packages/foo.pth.

At least this solution gives me a way to develop my libraries outside of site-packages.

Josh

 

[Josh English]

I have .egg files in my system path. The Egg file created by my setup script doesn't include anything but the introductory text. If I open other eggs I see the zipped data, but not for my own files.

Is having a zipped egg file any faster than a regular package? or does it just prevent people from seeing the code?

Josh

 

[Josh English]

I have .egg files in my system path. The Egg file created by my setup script doesn't include anything but the introductory text. If I open other eggs I see the zipped data, but not for my own files.

Is having a zipped egg file any faster than a regular package? or does it just prevent people from seeing the code?

Josh

 

[Josh English]

When I run: os.listdir('c:\Python27\lib\site-packages') I get the contents in order, so the folders come before .pth files (as nothing comes before something.) I would guess Python is using os.listdir. Why wouldn't it?

 

[Josh English]

When I run: os.listdir('c:\Python27\lib\site-packages') I get the contents in order, so the folders come before .pth files (as nothing comes before something.) I would guess Python is using os.listdir. Why wouldn't it?

 

[Stephen Hansen]

I have .egg files in my system path. The Egg file created by my setup script doesn't include anything but the introductory text. If I open othereggs I see the zipped data, but not for my own files.

Sounds like your setup.py isn't actually including your source.

Is having a zipped egg file any faster than a regular package? or does it just prevent people from seeing the code?

IIUC, its nominally very slightly faster to use an egg, because it can
skip a lot of filesystem calls. But I've only heard that and can't
completely confirm it (internal testing at my day job did not
conclusively support this, but our environments are uniquely weird).

But that speed boost (if even true) isn't really the point of
eggs-as-files -- eggs are just easy to deal with as files is all. They
don't prevent people from seeing the code*, they're just regular zip
files and can be unzipped fine.

I almost always install unzip my eggs on a developer machine, because I
inevitably want to go poke inside and see what's actually going on.

--

Stephen Hansen
... Also: Ixokai
... Mail: me+list/python (AT) ixokai (DOT) io
... Blog: http://meh.ixokai.io/

* Although you can make an egg and then go and remove all the .PY files
from it, and leave just the compiled .PYC files, and Python will load it
fine. At the day job, that's what we do. But, you have to be aware that
this ties the egg to a specific version of Python, and its not difficult
for someone industrious to disassemble and/or decompile the PYC back to
effectively equivalent PY files to edit away if they want.


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.10 (Darwin)

iQEcBAEBAgAGBQJOWYFsAAoJEKcbwptVWx/l16oIAIv6U0Y+wVP3TszeZ+Wfh3uz
IeE5A7g5XGsykKv/vPtTWEdepBdYIDvz4TUcpFKlfTLp64l//SPx1DDOpfrtjDJe
p5103k/o0P980Ke6ASKyIHndFQqJ/FQcsAnMHS/vIuahjOqCj2AF/FOCTRP35cvb
bZTucVsAEW7JsBo7LsJl2p1YySZt79DgHyWhY6lGPTVZQ3ye0t9dDegCF7YnK1kd
sSNwguFhjUh0OaNHcwr3eoUrWYPea3/695lJaGToYMAtqGEq4Y0EaxMgBMpo0oHa
aY0TUW9qzZEqm/MYn0DEj/aPTDXO7L94Tk60BSMwzo90syTdpCNI3KKRPB5Vx40=
=sArL
-----END PGP SIGNATURE-----

 

[Philip Semanchuk]

When I run: os.listdir('c:\Python27\lib\site-packages') I get the contents in order, so the folders come before .pth files (as nothing comes before something.)

That's one definition of "in order". =)

I would guess Python is using os.listdir. Why wouldn't it?

If you mean that Python uses os.listdir() during import resolution, then yes I agree that's probable. And os.listdir() doesn't guarantee any consistent order. In fact, the documentation explicitly states that the list is returned in arbitrary order. Like a lot of things in Python, os.listdir() probably relies on the underlying C library which varies from system to system. (Case in point -- on my Mac, os.listdir() returns things in the same order as the 'ls' command, which is case-sensitive alphabetical, files & directories mixed -- different from Windows.)

So if import relies on os.listdir(), then you're relying on arbitrary resolution when you have a .pth file that shadows a site-packages directory. Those rules will probably work consistently on your particular system, you're developing a habit around what is essentially an implementation quirk.

Cheers
Philip

 

[OKB (not okblacke)]

OKB,

The setup.py script created the egg, but not the .pth file. I
created that myself.

Thank you for clarifying about how .pth works. I know "redirect
imports" was the wrong phrase, but it worked in my head at the
time. It appears, at least on my system, that Python will find
site-packages/foo before it finds and reads site-packages/foo.pth.

At least this solution gives me a way to develop my libraries
outside of site-packages.

Well, I'm still not totally sure what your setup is, but assuming
site-packages/foo is a directory containing an __init__.py (that is, it
is a package), then yes, it will be found before an alternative package
in a directory named with a .pth file. Note that I don't say it will be
found before the .pth file, because, again, the finding of the package
(when you do "import foo") happens much later than the processing of the
..pth file. So it doesn't find site-packages/foo before it reads
foo.pth; it just finds site-packages/foo before it finds the other foo
that foo.pth was trying to point to.

Let's say your .pth file specifies the directory /elsewhere. The
..pth file is processed by site.py when the interpreter starts up, and at
that time /elsewhere will be appended to sys.path. Later, when you do
the import, it searches sys.path in order. site-packages itself will be
earlier in sys.path than /elsewhere, so a package site-packages/foo will
be found before /elsewhere/foo. The key here is that the .pth file is
processed at interpreter-start time, but the search for foo doesn't take
place until you actually execute "import foo".

If you want to make your /elsewhere "jump the line" and go to the
front, look at easy_install.pth, which seems to have some magic code at
the end that moves its eggs ahead of site-packages in sys.path. I'm not
sure how this works, though, and it seems like a risky proposition.


--
--OKB (not okblacke)
Brendan Barnwell
"Do not follow where the path may lead. Go, instead, where there is
no path, and leave a trail."
--author unknown