saved sys.path

[Alex Martelli]

I have a local directory that I want to include in my sys.path

How to save it permanently?

Add to the site-packages directory of your Python instalation a textfile
named for example bengt.pth (name doesn't matter, extension does, and
must be lowercase) with the line of text:

/path/to/the/directory/you/mant


Alex

 

[Bengt dePaulis]

I have a local directory that I want to include in my sys.path

How to save it permanently?

Regards
/Bengt

 

[Eric S. Johansson]

Add to the site-packages directory of your Python instalation a textfile
named for example bengt.pth (name doesn't matter, extension does, and
must be lowercase) with the line of text:

/path/to/the/directory/you/mant

one of the things I've been bothered with using Python for larger
applications is path manipulation. For example, how does one create the
search path necessary to find your own modules if you do not wish
pollute site-packages with your private code.

the path file looks like this might be a good solution in preference to
modifying sys.path in your code. Are there any scope or limitation
rules for this path modifier? Can this modification be made specific to
an application or is the new path visible to the entire world (which
reintroduces the namespace pollution problem).

---eric

 

[Alex Martelli]

one of the things I've been bothered with using Python for larger
applications is path manipulation. For example, how does one create the
search path necessary to find your own modules if you do not wish
pollute site-packages with your private code.

the path file looks like this might be a good solution in preference to
modifying sys.path in your code. Are there any scope or limitation
rules for this path modifier? Can this modification be made specific to
an application or is the new path visible to the entire world (which
reintroduces the namespace pollution problem).

The modifications to sys.path performed by .pth files are visible to all
applications. A single application may (however it chooses to do so)
find out or determine a path it wants only for its own imports, and
prepend it to sys.path, during its startup (entry-point script). On
Windows the popular choice is to stick stuff in the registry; on most
Unix-like systems "run-command files" are usually preferred. The Mac
(Mac OS X) has an extra possibility, since "an application" is a
directory (with extension .app) and the Mac has its own conventions
about what is where inside that directory. Python is not particularly
different, from the point of view of "where does an app find its own
configuration information", from most other languages.


Alex

 

[Eric S. Johansson]

The modifications to sys.path performed by .pth files are visible to all
applications. A single application may (however it chooses to do so)
find out or determine a path it wants only for its own imports, and ....historical record deleted
> Python is not particularly
different, from the point of view of "where does an app find its own
configuration information", from most other languages.

from historical perspective, I understand and appreciate your
explanation. However, just because it's historically accurate, does it
mean we should keep doing things the same old way?

In natlink, Joel Gould's python extensions to NaturallySpeaking, he made
each speech recognition grammar environment either globally accessible
or command specific. He used the expedient technique of using the same
name as the command for the Python file containing the grammar
associated with that command. He also used another simple technique,
prefacing a filename with an "_" to indicate a globally visible grammar
and code. Turns out this was a frightfully useful technique as it
allows for partitioning grammars. Unfortunately, because of path
problems it's difficult to share modules between various grammars.

It seems to me that a similar technique would serve for our purposes. A
file, command_name.path, would contain all of the search as necessary
for a command called by the same name. this technique would allow for a
common platform independent method of finding application-specific
modules and automatically generated paths for site-specific
relocation/configuration.

As I write this, it occurs to me that a provisional body of code to
experiment with the technique would possibly serve the community well.
I will generate something tomorrow for public scrutiny as I am up past
my bedtime. Thank you for triggering the idea.

 

[Alex Martelli]

file, command_name.path, would contain all of the search as necessary
for a command called by the same name. this technique would allow for a
common platform independent method of finding application-specific
modules and automatically generated paths for site-specific
relocation/configuration.

As I write this, it occurs to me that a provisional body of code to
experiment with the technique would possibly serve the community well.
I will generate something tomorrow for public scrutiny as I am up past
my bedtime. Thank you for triggering the idea.

You're welcome. Implementation should be easy, if you think that
architecture is sound. Consider site.py, specifically the end of
function addsitedir(sitedir), which currently goes:

names.sort()
for name in names:
if name[-4:] == os.extsep + "pth":
addpackage(sitedir, name)
if reset:
_dirs_in_sys_path = None

It seems to me that all you need would be to make it:

names.sort()
for name in names:
if name[-4:] == os.extsep + "pth" or name == specname:
addpackage(sitedir, name)
if reset:
_dirs_in_sys_path = None

where specname is the application-specific path-filename you envisage.
At module-global level, where now you have:

prefixes = [sys.prefix]
sitedir = None # make sure sitedir is initialized because of later 'del'

your would add one further line going, for example:

specname = '%s.appath' % os.path.basename(sys.argv[0])


I would suggest .appath or something that stands out more than .path
which looks too much like .pth -- just IMHO.

Moreover, I think addpackage should take an optional flag:

def addpackage(sitedir, name, athead=False):

and use it when it adds to the path: instead of

if not dircase in _dirs_in_sys_path and os.path.exists(dir):
sys.path.append(dir)

it might go something like:

if not dircase in _dirs_in_sys_path and os.path.exists(dir):
if athead:
sys.path.insert(0, dir)
else:
sys.path.append(dir)

Then, the call to addpackage from functon addsite should become:

if name[-4:] == os.extsep + "pth" or name == specname:
addpackage(sitedir, name, name == specname)

All this is untested, but we're talking about a few lines worth of patch
to site.py, nothing huge.


As to whether the whole architecture is sound -- I pass. I'm not sure I
see why it makes sense to have a foo.appath (or foo.path as you'd want
it) that presumably gets autoedited during install to contain the
relevant dirs, rather than just autoedit foo.py, the main script, whose
purpose should, after all, be exactly to set up the environment
(sys.path first and foremost) before importing and running the real
stuff. The current bar.pth arrangement makes sense exactly because it's
_not_ tied to apps, but to sites -- where has this site chosen to
install some backage 'bar' which must be made available for import to
all scripts running at this site. An application, as opposed to a
package, gets to run its own code ("main script") and thus would seem
not to need such arrangements.


Alex

 

[Peter Hansen]

one of the things I've been bothered with using Python for larger
applications is path manipulation. For example, how does one create the
search path necessary to find your own modules if you do not wish
pollute site-packages with your private code.

the path file looks like this might be a good solution in preference to
modifying sys.path in your code. Are there any scope or limitation
rules for this path modifier? Can this modification be made specific to
an application or is the new path visible to the entire world (which
reintroduces the namespace pollution problem).

Most questions relating to this which haven't already been answered
by Alex can be answered by a perusal of the site.py source. It
makes good readin'...

Note that between PYTHONPATH and .pth files, possibly the addition
of a little sitecustomize.py file (see again site.py), and the
ability to modify sys.path in the __main__ module of the app as
Alex suggests, you can do pretty much anything you think is a good
thing to do.

-Peter

 

[Eric S. Johansson]

You're welcome. Implementation should be easy, if you think that
architecture is sound. Consider site.py, specifically the end of
function addsitedir(sitedir), which currently goes:

....

hmm you must have a later bedtime than me. I was thinking a slightly
different direction in terms of integration etc. but these are good
ideas too and I will lift with appropriate credits.

I would suggest .appath or something that stands out more than .path
which looks too much like .pth -- just IMHO.

true. I have come to accept the fact that I should never be allowed to
name anything except the occasional pet. Personally I preferred an
extension that is speech recognition friendly since all of these mangled
names really hurt to type and are incredibly difficult to spell out
phonetically given current speech recognition accuracy's for nonhuman
language text sequences.

As to whether the whole architecture is sound -- I pass. I'm not sure I
see why it makes sense to have a foo.appath (or foo.path as you'd want
it) that presumably gets autoedited during install to contain the
relevant dirs, rather than just autoedit foo.py, the main script, whose
purpose should, after all, be exactly to set up the environment
(sys.path first and foremost) before importing and running the real
stuff. The current bar.pth arrangement makes sense exactly because it's
_not_ tied to apps, but to sites -- where has this site chosen to
install some backage 'bar' which must be made available for import to
all scripts running at this site. An application, as opposed to a
package, gets to run its own code ("main script") and thus would seem
not to need such arrangements.

from my perspective, it makes a great deal of sense. If you are making
modules for general use, site-packages is a great place to deposit your
work. However, if you are building applications which have collections
of modules that need to be imported, site-packages is not the place to
put them especially if there is a risk of a namespace collision.

I will confess to not being very familiar with the foo.py model you
describe and references would be welcome. my philosophical base is one
of separation of code and data. Which means that configuration data
such as search paths etc. belong "somewhere else". Also to modify a
program file even when it does nothing but set up the context, makes me
really uncomfortable from a environment management and security
perspective. That's awfully close to what a worm or a virus would do as
part of its reproductive imperative. although, even I would admit that
a python based worm would probably starve to death...

so anything I can do to keep code static and as identical as possible to
what was dropped out of darcs yet still be flexible enough to
accommodate different installation locations, I consider a win.

we have a couple philosophies that work for different people. It would
be nice to have enough room to include both.

---eric

PS, here's hoping for a clear regime change here in the states. Wish us
luck.

 

[Scott David Daniels]

from historical perspective, I understand and appreciate your
explanation. However, just because it's historically accurate, does it
mean we should keep doing things the same old way?

There is a danger to watch out for here. A great Python success is that
it works in and on the local OS. I have always felt that languages as
otherwise successful as APL and Smalltalk have failed by defining their
own conventions (for file storage and such), rather than respecting the
local conventions.

Python attempts the local dance, and it is locality, not history, that
affects the details of the dance. Mac people (for example) won't care
about the "Python convention" initially and might reject Python because
"it doesn't put information in the right place." There is a line here
to be careful about. History is not an arrow, but an exploration of
possibilities.

I mention this not as a "killer rebuttal," but as a "there be dragons here."

-Scott David Daniels
(e-mail address removed)

 

[Eric S. Johansson]

Alex Martelli wrote:
....bunch of good stuff

here's my working model. It's barely tested and I'll experiment with it
in some code of the next couple of days.

its usage is relatively simple. Stick this bit of code in
site-packages. I call it application_path.py because it's easy to
dictate. Then in a piece of code such as xyzzy.py, you start it was
something like:

#!/usr/bin/python

import application_path
application_path.set_path()
import sys
print sys.path

and create xyzzy.pathlist in site-packages. Fill it with a list of
directories that exist. If directories don't exist, they will not be
added to the search path. And run the executable. The print statement
will show you modified path. In this example, the default is to prepend
paths because in my world, I want my changes to be found first just in
case there is a naming conflict or I am overriding a system module.

the first argument to set_path is the name. specifying the name is
useful obviously because you may have multiple executables that use the
same module set and this allows the application set to use a single path
file. The second is at_head which if true (default) places the paths at
the beginning of sys.path. If false, obviously they are appended to the
end.

as for the code style, it's mostly influenced by what is easy to do
using speech recognition and Emacs. There have been a couple of
projects to try and make programming by voice easier but they have run
out of steam or run over by complexity. Maybe both.


#!/usr/bin/python

#application_path is a module an application can use to add
#application-specific paths to an application's awareness

import sys
import os.path
import re

def set_path(name = None, at_head = True):
"""expand the site path from the file matching the given name or
the executable name. Always append to the head of the search path
unless at_head == False"""

working_name = name
if name == None:
# strip off suffix and/or leading path
working_name = sys.argv[0]

# print "proto working name %s"% working_name
dot_index = working_name.rfind(".")
if dot_index != -1:
working_name = working_name[0:dot_index]
working_name = os.path.basename(working_name)

# convert name to path list
path_list = get_paths(working_name)

if at_head:
path_list.extend(sys.path)
sys.path=path_list

else:
sys.path.extend(path_list)

print sys.path
return working_name


def get_paths (name):
"""based on a given name (which should be fully cleaned up to be
just a name and not a path with extension) get a path file and """
file_path = "".join([sys.prefix,
"/lib/python",
sys.version[:3],
"/site-packages/",
name,
".pathlist",
]
)
try:
#print "file_path %s" % file_path
handle = open(file_path, 'r')

path_list = []
for line in handle.readlines():
line = line.strip()

#ignore comments and blank lines
ignore = re.search(r'^#|^\s+$', line)
if ignore : continue

# does path exist?
if os.path.isdir(line):

# build a list of paths
path_list.append(line.strip())

handle.close()

return path_list

except IOError:
#print "exception: file_path %s" % file_path
return []


if __name__ == '__main__':

set_path()
print
print

set_path("testing")

print
print

set_path("testing", False)

 

[Eric S. Johansson]

Most questions relating to this which haven't already been answered
by Alex can be answered by a perusal of the site.py source. It
makes good readin'...

I'm sure it does and I will spend some bedtime reading (after I finish
"lost in translation", an extremely good book that only shares the title
with a movie. I also recommend tsil cafe.)

Note that between PYTHONPATH and .pth files, possibly the addition
of a little sitecustomize.py file (see again site.py), and the
ability to modify sys.path in the __main__ module of the app as
Alex suggests, you can do pretty much anything you think is a good
thing to do.

obviously, I need to grok site.py more. But as a basic philosophical
basis, I really believe that configuration information should be
externalized from the program as much as humanly possible. So things
like modifying sys.path in __main__ is a source of maintenance problems
further down the road. the way I've been able to encapsulate the
changes in my demonstration code and separate configuration file treats
for me an environment which is much more predictable and reliable.

Another example is what I've done with configuration files in the camram
project. I created a configuration file module which uses three files
to build a single view of configuration data. At the base is the
distribution default configuration elements. Overlaid on that is the
site configuration file data and last overlaid is the user specific
configuration file data. End result is that one I upgrade the system, I
never ever need to touch the configuration file unless I'm modifying a
new default.

Another thing I did in order to keep myself from making lots of mistakes
is that I prefaced every string in the configuration file with a type
information and as a result when the configuration file variable is
returned, is always returned in the right form. In addition to the
normal string, integer, floats, I've also created paths so that file
names will be automatically converted to the right form for that host.

I guess this is a long-winded way of saying that I try to create
environmental support to keep me from making mistakes in my code. I
forget things, I use speech recognition and have to deal with
recognition errors in addition to thinking about code or writing. It's
a helluva lot of cognitive effort so if I can make my code and modules
work for me and catch my mistakes automatically, then it's a win.

but I will take your advice to heart. Thank you for giving it.

---eric

 

[Alex Martelli]

basis, I really believe that configuration information should be
externalized from the program as much as humanly possible. So things

I agree. But putting it in a scriptfile instead of a datafile is just
as good, IMHO -- just like, say, config info for my bash shell sessions
goes into a file .bashrc which is a script rather than being just a
datafile. Most of what .bashrc does is setting environment variables,
but the extra flexibility of it being a script may still help.

Similarly, as I see things, config info for a big app (coded in Python,
or in whatever other language as long as it's got a Python accessible
interface) can be in a scriptfile instead of a datafile. The scriptfile
starts up, adjusts environment info (sys.path foremost, but maybe other
stuff too), imports the app's main module, launches 'main(sys.argv)'
therein. It _is_ externalized from the program, in my view -- it's a
configuration scriptfile. It just works better, with Python, to have
the configuration scriptfile call the app's main, rather than doing it
the other way round as you would in, say, C (where the app's main starts
and then calls a configuration scriptfile).

I do like what you've posted, don't get me wrong -- I'm just pointing
out that our different preferences on app configuration architectures do
not descend from differences on the "externalized from the program"
philosophy!


Alex

 

[Eric S. Johansson]

I agree. But putting it in a scriptfile instead of a datafile is just
as good, IMHO -- just like, say, config info for my bash shell sessions
goes into a file .bashrc which is a script rather than being just a
datafile. Most of what .bashrc does is setting environment variables,
but the extra flexibility of it being a script may still help.

agreed to a point. What makes me uncomfortable about configuration
files like .mumblerc files is what happens to them over time. That this
is not a python specific problem and it may just be a quirk of my
personality.

over time, the base knowledge underlying configuration files changes and
manual handling of those configuration files on every update should not
be a requirement. automatically updating files which are scripts as
well as configuration data is much more difficult than just
configuration files alone.

Another irritation is that if I set up configuration files to create a
particular environment on one machine, replicating the machine
independent portions of those configuration files and keeping everything
else up-to-date spread across a 10 or 15 other machines is quite a pain.
it really could be handled better. Not quite sure how yet but I'm
thinking about it. The solution is starting to smell like starter kit
and Web server containing machine specific and independent configuration
information.

Similarly, as I see things, config info for a big app (coded in Python,
or in whatever other language as long as it's got a Python accessible
interface) can be in a scriptfile instead of a datafile. The scriptfile
starts up, adjusts environment info (sys.path foremost, but maybe other
stuff too), imports the app's main module, launches 'main(sys.argv)'
therein. It _is_ externalized from the program, in my view -- it's a
configuration scriptfile. It just works better, with Python, to have
the configuration scriptfile call the app's main, rather than doing it
the other way round as you would in, say, C (where the app's main starts
and then calls a configuration scriptfile).

so let me see if I understand. If a application has X commands, there
are X script files which set up the appropriate environments and call
the individual command mainline routines?

it would probably help me if you could point me at an example of which
you would consider a good version of this environment.

I do like what you've posted, don't get me wrong -- I'm just pointing
out that our different preferences on app configuration architectures do
not descend from differences on the "externalized from the program"
philosophy!

thank you. It could probably be included into site.py without too much
trouble if there was any interest. I've been meaning to submit for
consideration a couple things including a version of smtpd.py that forks
off child processes and a file based queue system. But I fear this
conversation would start heading in the direction of the cpyan thread.

---eric