ANN: New configuration module released

[Vinay Sajip]

I've just made available the first general release of a configuration
module for Python. It has grown out of the need to make configuration
easier and more powerful (for developers and end users) than
ConfigParser.

It's one of the implementations which could be considered in the
ConfigParser shootout, mentioned on the Python Wiki at

http://www.python.org/moin/ConfigParserShootout

What Does It Do?
================
The config module allows you to implement a hierarchical configuration
scheme with support for mappings and sequences, cross-references
between one part of the configuration and another, the ability to
flexibly access real Python objects, facilities for configurations to
include and cross-reference one another, simple expression evaluation
and the ability to change, save, cascade and merge configurations.

This module has been developed on python 2.3 but should work on
version 2.2 or greater. A test suite using unittest is included in the
distribution.

A very simple configuration file (simple.cfg):

# starts here
message: Hello, world!
#ends here

a very simple program to use it:

from config import Config

cfg = Config(file('simple.cfg'))
print cfg.message

results in:

Hello, world!

Configuration files are key-value pairs, but the values can be
containers that contain further values
A simple example - with the example configuration file:

messages:
[
{
stream : `sys.stderr`
message: 'Welcome'
name: 'Harry'
}
{
stream : `sys.stdout`
message: 'Welkom'
name: 'Ruud'
}
{
stream : $messages[0].stream
message: 'Bienvenue'
name: Yves
}
]

a program to read the configuration would be:

from config import Config

f = file('simple.cfg')
cfg = Config(f)
for m in cfg.messages:
s = '%s, %s' % (m.message, m.name)
try:
print >> m.stream, s
except IOError, e:
print e

which, when run, would yield the console output:

Welcome, Harry
Welkom, Ruud
Bienvenue, Yves

The above example just scratches the surface. There's more information
about this module available at

http://www.red-dove.com/python_config.html

Comprehensive API documentation is available at

http://www.red-dove.com/config/index.html

As always, your feedback is most welcome (especially bug reports,
patches and suggestions for improvement). Enjoy!

Cheers

Vinay Sajip
Red Dove Consultants Ltd.

 

[Dan Perl]

Vinay,

I'm posting some opinions I have on your module and on the topic in general
in this newsgroup, instead of using the links you have on your web page
(your email and a wiki) because I would like to see a wider discussion.

My experience with python in general and configuration in python in
particular is rather short (a few months), but I have been working on a
personal, open-source project during this time and configuration was one of
the biggest issues I had. I did look at configParser early on, but I
decided it was not giving me enough. At the top level, I had only a few
values (4 at present, but even less in the beginning) and these values were
however fairly complex. I ended up implementing the configuration mechanism
as plain python modules which are imported by my application. This created
a mechanism that was relatively difficult to understand and that needed
users to actually code in python for configuration. I think the problem of
configuration in my project is much smaller now, because I also added a GUI
tool that allows configuration to be done through a graphical interface.

Enough about my project, but I meant to give a context for what I am about
to discuss next.

I looked only at the documentation for your module and I haven't actually
tried it. Please forgive me if I am missing anything. With this
information though, I don't see enough reasons to replace my own custom-made
configuration mechanism with your module.

First, some small issues. The syntax for your configuration is still fairly
close to the python syntax and it involves quite a bit of learning for both
python users and non-python users. Your module would save me a little bit
of work in parsing when loading a configuration (remember, I am loading
configurations by importing them as a module) and a little more work than
that in formatting when saving a configuration. That work is not very
significant though and having my own mechanism also has its advantages.

Now, here is the biggest issue I have and what I would be extremely
interested in seeing in a configuration module. I don't see from your
documentation how I could programmatically create a configuration from
scratch. In my project I have a set of handlers and each handler has a
special configuration. Each such special configuration is defined by a
structure of metadata (with information like name, type, description) that I
implemented myself. That way, my GUI tool can load a handler dynamically,
read its configuration metadata, dynamically create a frame for the
handler's configuration, and save the new configuration to a file.

Again, I only read your documentation, but I haven't seen anything like my
configuration metadata, that would allow me to dynamically create a
configuration from scratch. Please correct me if I'm wrong and point me to
something that your module has for this purpose.

I definitely appreciate your effort though and I hope that you continue
working on this configuration module. Like I said, configuration has been a
big issue in my project and I would like to have a generic configuration
that would satisfy all my needs instead of creating (and now evolving and
maintaining) my own, custom-made configuration mechanism. Something like
the configuration metadata that I described is a very important feature for
me. I hope I described it clearly enough. And again, my apologies if you
already have such a feature and I only missed it.

Dan

I've just made available the first general release of a configuration
module for Python. It has grown out of the need to make configuration
easier and more powerful (for developers and end users) than
ConfigParser.

It's one of the implementations which could be considered in the
ConfigParser shootout, mentioned on the Python Wiki at

http://www.python.org/moin/ConfigParserShootout

What Does It Do?
================
The config module allows you to implement a hierarchical configuration
scheme with support for mappings and sequences, cross-references
between one part of the configuration and another, the ability to
flexibly access real Python objects, facilities for configurations to
include and cross-reference one another, simple expression evaluation
and the ability to change, save, cascade and merge configurations.

This module has been developed on python 2.3 but should work on
version 2.2 or greater. A test suite using unittest is included in the
distribution.

A very simple configuration file (simple.cfg):

# starts here
message: Hello, world!
#ends here

a very simple program to use it:

from config import Config

cfg = Config(file('simple.cfg'))
print cfg.message

results in:

Hello, world!

Configuration files are key-value pairs, but the values can be
containers that contain further values
A simple example - with the example configuration file:

messages:
[
{
stream : `sys.stderr`
message: 'Welcome'
name: 'Harry'
}
{
stream : `sys.stdout`
message: 'Welkom'
name: 'Ruud'
}
{
stream : $messages[0].stream
message: 'Bienvenue'
name: Yves
}
]

a program to read the configuration would be:

from config import Config

f = file('simple.cfg')
cfg = Config(f)
for m in cfg.messages:
s = '%s, %s' % (m.message, m.name)
try:
print >> m.stream, s
except IOError, e:
print e

which, when run, would yield the console output:

Welcome, Harry
Welkom, Ruud
Bienvenue, Yves

The above example just scratches the surface. There's more information
about this module available at

http://www.red-dove.com/python_config.html

Comprehensive API documentation is available at

http://www.red-dove.com/config/index.html

As always, your feedback is most welcome (especially bug reports,
patches and suggestions for improvement). Enjoy!

Cheers

Vinay Sajip
Red Dove Consultants Ltd.

 

[Neal D. Becker]

Looks good so far.

The boost c++ library (about to be released) has a new program_options
parser. One feature I appreciate in the design, is that the source of
options is seperated from the rest of the design. This means that, out of
the box, it reads options from command line, from config files, and from
env variables. Probably not hard to extend if anyone could think of a
reason.

I believe a decent config module should at least handle both config files
and command line options. ENV variables are a nice touch too.

 

[Vinay Sajip]

Dan,

I'm posting some opinions I have on your module and on the topic in general
in this newsgroup, instead of using the links you have on your web page
(your email and a wiki) because I would like to see a wider discussion.
Fine.

First, some small issues. The syntax for your configuration is still fairly
close to the python syntax and it involves quite a bit of learning for both
python users and non-python users. Your module would save me a little bit
of work in parsing when loading a configuration (remember, I am loading
configurations by importing them as a module) and a little more work than
that in formatting when saving a configuration. That work is not very
significant though and having my own mechanism also has its advantages.

One important benefit of having Python-like syntax is that it is easy to
create a configuration programmatically, using just dicts and lists.
This is something that you cite below as a big issue for you.

In terms of learning - I'm not sure that python developers would have
any real problem with the basic syntax. For end users, it's a case of
unfamiliarity rather than intrinsic difficulty, IMO.

Now, here is the biggest issue I have and what I would be extremely
interested in seeing in a configuration module. I don't see from your
documentation how I could programmatically create a configuration from
scratch. In my project I have a set of handlers and each handler has a
special configuration. Each such special configuration is defined by a
structure of metadata (with information like name, type, description) that I
implemented myself. That way, my GUI tool can load a handler dynamically,
read its configuration metadata, dynamically create a frame for the
handler's configuration, and save the new configuration to a file.

There are many ways you could do what you want to do. Without knowing
the details of your application, I can't give particularly good advice.
However, consider the following program:

from config import Config
from cStringIO import StringIO

root = { }
messages = [ ]
root['total_period'] = 100
root['test'] = True
root['another_test'] = False
root['yet_another_test'] = None
root['messages'] = messages
root['message'] = 'Hello, world!'
messages.append({ 'message' : 'Welcome', 'name' : 'Harry' })
messages.append({ 'message' : 'Welkom', 'name' : 'Ruud' })
messages.append({ 'message' : 'Bienvenue', 'name' : 'Yves' })

stream = StringIO()
print >> stream, root
value = stream.getvalue()
stream.close()

stream = StringIO(value[1:-2])
cfg = Config(stream)
cfg.save(file('test.txt', 'w'))

This shows how easy it is to programmatically construct a simple
configuration from scratch.

It would be easy for your configuration to hold a separate tree in the
configuration with the metadata. For example, you could construct a GUI
frame given a set of fields:

fields: [ title, firstname, surname, age, jobtitle ]
metadata: {
title : { description: 'Salutation', type: choice,
values: [ [1, 'Mr.'], [2, 'Mrs.']] }
firstname : { description: 'First name', type: text,
max_length: 20 }
surname : { description: 'Last name', type: text, max_length: 20 }
age : { description: 'Age', type: int }
jobtitle: { description: 'Job title', type: text }
}

Then you can easily loop through the fields.

from config import Config

cfg = Config(file('metadata.cfg'))

for fld in cfg.fields:
metadata = cfg.get('metadata.' + fld)
print "name = %r, type = %r, description = %r" %
(fld, metadata.type, metadata.description)

with the result

name = 'title', type = 'choice', description = 'Salutation'
name = 'firstname', type = 'text', description = 'First name'
name = 'surname', type = 'text', description = 'Last name'
name = 'age', type = 'int', description = 'Age'
name = 'jobtitle', type = 'text', description = 'Job title'

I hope the above allows you to see the potential that the config module
has to meet your needs.

Best wishes,


Vinay Sajip

 

[Vinay Sajip]

Looks good so far.

The boost c++ library (about to be released) has a new program_options
parser. One feature I appreciate in the design, is that the source of
options is seperated from the rest of the design. This means that, out of
the box, it reads options from command line, from config files, and from
env variables. Probably not hard to extend if anyone could think of a
reason.

I believe a decent config module should at least handle both config files
and command line options. ENV variables are a nice touch too.

Python already has the optparse module for command-line options, and
it's pretty good. I'll be looking at ways in which access to
command-line options parsed via optparse can be made available through
the config API, without duplication of code.

Regards,


Vinay Sajip

 

[Dan Perl]

One important benefit of having Python-like syntax is that it is easy to
create a configuration programmatically, using just dicts and lists.
This is something that you cite below as a big issue for you.

In terms of learning - I'm not sure that python developers would have
any real problem with the basic syntax. For end users, it's a case of
unfamiliarity rather than intrinsic difficulty, IMO.

You're right, but my point is that for python programmers it's even better
if the configuration is implemented in plain python instead of using a
python-like syntax. As for non-python programmers, which is where my
biggest concern was, both solutions are an inconvenience.

As for creating a configuration programmatically, I can do that using python
for the syntax (and I am already doing that). Your syntax is better for
other, non-python applications that may also be interested in using these
configuration mechanism, but that is not important for me. Besides,
non-python applications would still have to implement their own parsing and
formatting for this syntax.

It's just an idea and I haven't put a lot of thought in it myself, but what
about using XML for the configuration syntax?

Now, here is the biggest issue I have and what I would be extremely
interested in seeing in a configuration module. I don't see from your
documentation how I could programmatically create a configuration from
scratch. In my project I have a set of handlers and each handler has a
special configuration. Each such special configuration is defined by a
structure of metadata (with information like name, type, description)
that I implemented myself. That way, my GUI tool can load a handler
dynamically, read its configuration metadata, dynamically create a frame
for the handler's configuration, and save the new configuration to a
file.

There are many ways you could do what you want to do. Without knowing
the details of your application, I can't give particularly good advice.
However, consider the following program:

from config import Config
from cStringIO import StringIO

root = { }
messages = [ ]
root['total_period'] = 100
root['test'] = True
root['another_test'] = False
root['yet_another_test'] = None
root['messages'] = messages
root['message'] = 'Hello, world!'
messages.append({ 'message' : 'Welcome', 'name' : 'Harry' })
messages.append({ 'message' : 'Welkom', 'name' : 'Ruud' })
messages.append({ 'message' : 'Bienvenue', 'name' : 'Yves' })

stream = StringIO()
print >> stream, root
value = stream.getvalue()
stream.close()

stream = StringIO(value[1:-2])
cfg = Config(stream)
cfg.save(file('test.txt', 'w'))

This shows how easy it is to programmatically construct a simple
configuration from scratch.

I probably didn't express clearly enough what I need in my initial posting.
Your example is for a statically defined configuration structure. I am
interested in a configuration structure that can be defined dynamically. So
if I load a plugin with a configuration structure that I don't know
beforehand, I can still create that configuration programmatically.

It would be easy for your configuration to hold a separate tree in the
configuration with the metadata. For example, you could construct a GUI
frame given a set of fields:

fields: [ title, firstname, surname, age, jobtitle ]
metadata: {
title : { description: 'Salutation', type: choice,
values: [ [1, 'Mr.'], [2, 'Mrs.']] }
firstname : { description: 'First name', type: text,
max_length: 20 }
surname : { description: 'Last name', type: text, max_length: 20 }
age : { description: 'Age', type: int }
jobtitle: { description: 'Job title', type: text }
}

Then you can easily loop through the fields.

from config import Config

cfg = Config(file('metadata.cfg'))

for fld in cfg.fields:
metadata = cfg.get('metadata.' + fld)
print "name = %r, type = %r, description = %r" %
(fld, metadata.type, metadata.description)

with the result

name = 'title', type = 'choice', description = 'Salutation'
name = 'firstname', type = 'text', description = 'First name'
name = 'surname', type = 'text', description = 'Last name'
name = 'age', type = 'int', description = 'Age'
name = 'jobtitle', type = 'text', description = 'Job title'

This is a great idea and it's very useful for me: defining the metadata as a
configuration file. And your configuration module could be useful for that.
Thanks. I will consider it.

And yet, I would still like to see a generic mechanism for configuration
metadata. And this is the point where I expected to see a discussion. Am I
the only one seeing a need to define configuration structures with metadata?
I see that useful for:
- dynamically defined configurations like in my project
- sharing the same configuration structure between several applications,
so
for instance a configuration created by one application can be used by
another application

Any opinions on that?

Dan

 

[Vinay Sajip]

You're right, but my point is that for python programmers it's even better
if the configuration is implemented in plain python instead of using a
python-like syntax. As for non-python programmers, which is where my
biggest concern was, both solutions are an inconvenience.

OK, but I'm not sure that completely unrestricted Python code in
configurations is the way to go. It certainly makes it more likely that
a developer will put things in the configuration which an end user can
inadvertently break. If it's pure Python in the configuration, it's
possible that the breakage will cause more damage than where a
Python-like syntax is parsed.

As for creating a configuration programmatically, I can do that using python
for the syntax (and I am already doing that). Your syntax is better for
other, non-python applications that may also be interested in using these
configuration mechanism, but that is not important for me. Besides,
non-python applications would still have to implement their own parsing and
formatting for this syntax.

It may not be important to you, but that's one reason why XML is so
popular: data can be shared across platforms more easily.

It's just an idea and I haven't put a lot of thought in it myself, but what
about using XML for the configuration syntax?

It's certainly possible, but very verbose for both reading and writing,
and more error-prone if end-users can change configuration files. It's
easy to make an XML file ill-formed through simple typing errors, like
forgetting the / to terminate an empty tag - even experienced developers
do this. Sometimes an XML parser can report the error several lines
further down, depending on the DTD/schema being used.

I probably didn't express clearly enough what I need in my initial posting.
Your example is for a statically defined configuration structure. I am
interested in a configuration structure that can be defined dynamically. So
if I load a plugin with a configuration structure that I don't know
beforehand, I can still create that configuration programmatically.

I only gave a simple example to illustrate possibilities. There's no
reason why you couldn't do this with a more dynamic structure - that's
the province of your program, not the config module.

This is a great idea and it's very useful for me: defining the metadata as a
configuration file. And your configuration module could be useful for that.
Thanks. I will consider it.
Great.

And yet, I would still like to see a generic mechanism for configuration
metadata. And this is the point where I expected to see a discussion. Am I
the only one seeing a need to define configuration structures with metadata?
I see that useful for:
- dynamically defined configurations like in my project
- sharing the same configuration structure between several applications,
so
for instance a configuration created by one application can be used by
another application

The problem I see is that everyone's idea of metadata will be different.
If you have strong and rigorous requirements in this area, the rigour
of XML and XML schema may be what you need. It's not a goal of the
config module to be all things to all people - it's meant to remain
reasonably simple while still providing useful functionality, enough to
supplant ConfigParser for those who need it.

Best regards,


Vinay Sajip

 

[Vinay Sajip]

Neal,

I believe a decent config module should at least handle both config files
and command line options. ENV variables are a nice touch too.

Further to my earlier post, I did a bit of thinking and find it very
easy to accommodate command line options read via optparse within the
overall configuration framework. Consider the following hypothetical
configuration file 'cmdline.cfg':

cmdline_values:
{
verbose : `cmdline.verbose`
file: `cmdline.filename`
}
other_config_items:
{
whatever : 'you want'
}

The values of interest are those provided in the 'cmdline_values' part
of the configuration. (You can, of course, use whatever structure you
like - this is only an example.)

Consider the following program:

from optparse import OptionParser
from config import Config

parser = OptionParser()
parser.add_option("-f", "--file",
action="store", type="string", dest="filename",
help="write report to FILE", metavar="FILE")
parser.add_option("-q", "--quiet",
action="store_false", dest="verbose", default=1,
help="don't print status messages to stdout")

(options, args) = parser.parse_args()

cfg = Config(file('cmdline.cfg'))
cfg.addNamespace(options, 'cmdline')
print "The verbose option value is %r" % cfg.cmdline_values.verbose
print "The file name is %r" % cfg.cmdline_values.file

After parsing the command line using OptionParser, we read in the
configuration (possibly getting the configuration file from the parsed
options, though not in this case). We also add a namespace called
'cmdline' which adds the parsed option settings to the configuration.
When we later reference cfg.cmdline_values.verbose, this references
`cmdline.verbose`, and this resolves to the 'verbose' attribute of the
parsed-options object (added under the name 'cmdline'). The program will
print

The verbose option value is False
The file name is 'test'

as you would expect.

N.B. I had to make a small change to the Config.addNamespace method to
get this to work. The current release (0.3) does not have this fix, but
the next release will.

Best regards,



Vinay Sajip

 

[Neal Becker]

Python already has the optparse module for command-line options, and
it's pretty good. I'll be looking at ways in which access to
command-line options parsed via optparse can be made available through
the config API, without duplication of code.

The point is, you would like to specify the options once, and use this
single specification for all option sources - instead of redundantly having
to specify that there is a 'verbosity' option that takes a value of 'int'
and has a default of 'whatever', and do this once (and with different
syntax) for each source - config files, command line, etc.

 

[Vinay Sajip]

The point is, you would like to specify the options once, and use this
single specification for all option sources - instead of redundantly having
to specify that there is a 'verbosity' option that takes a value of 'int'
and has a default of 'whatever', and do this once (and with different
syntax) for each source - config files, command line, etc.

The optparse module is very good for parsing command line options, but
is not a general purpose program configuration facility. It's not really
sensible to grow optparse to become a general purpose program
configuration module.

The config module aspires to be a good configuration facility, but does
not concern itself with parsing command line options. There's little
point in reinventing the wheel when optparse is more than fit for
purpose, for many people.

Hence, the best option is to make them work together, but it's unlikely
that you'll get a 'one size fits all'. For example, some applications
will allow a user to specify which configuration to load via command
line options. In theory, a configuration file could be used to configure
command line options and you could call optparse with those. But it's
chicken and egg in many scenarios, and command line options for a
program tend to be less volatile than configurations in general. I think
it's more natural to specify configurations in a configuration file, and
allow overriding some subset of them via command-line arguments.

I've posted (in another post) an example of how config and optparse can
work together with minimal fuss.

Best regards,


Vinay Sajip

 

[Hung Jung Lu]

I've just made available the first general release of a configuration
module for Python.

I know you probably want a lot of features, but then different people
want different things. What a person considers necessary, others may
find them unnecessary, and vice versa.

Microsoft went from .INI files to Windows registry, and now to
XML-based app.config files in the .NET world. I know app.config files
are not the best thing with the cleanest format, but, why re-invent
the wheel?

Microsoft's XML-based app.config is guaranteed to be widely used. For
something as minor as configuration files, for people that juggle
between many programming languages, I guess it's an OK solution.

I know many people may not like the XML overhead, I know many people
don't like Microsoft. But when you have to use hierarchical
configuration files across many programming languages, I don't know
any format out there that can compete with app.config. To me, the war
is already over.

regards,

Hung Jung

 

[Sylvain Thenault]

Looks good so far.

The boost c++ library (about to be released) has a new program_options
parser. One feature I appreciate in the design, is that the source of
options is seperated from the rest of the design. This means that, out of
the box, it reads options from command line, from config files, and from
env variables. Probably not hard to extend if anyone could think of a
reason.

I believe a decent config module should at least handle both config files
and command line options. ENV variables are a nice touch too.

the "configuration" module in logilab's common library is doing this (make
a bridge between optparse and ConfigParser). You give a kind of
configuration schema to a configuration object and then it can be
initialized from a configuration file and/or command line. Environment
variable are not considered yet, but I don't think it would be hard to add
this feature. The main problem is the lack of documentation, but if there
is some interest I can post an usage example.

Notice that this library also provides some new option types for optparse
(usable by the configuration module, of course), such as regexp (for
regular expression), csv (comma separated values), yn (yes or no) and
named (arbitrary name=value pair).

 

[Vinay Sajip]

the "configuration" module in logilab's common library is doing this (make
a bridge between optparse and ConfigParser). You give a kind of
configuration schema to a configuration object and then it can be
initialized from a configuration file and/or command line. Environment
variable are not considered yet, but I don't think it would be hard to add
this feature. The main problem is the lack of documentation, but if there
is some interest I can post an usage example.

I'm interested to see how your bridging works.

Using the config module I released, access to environment variables is
very easy. Here's how:

1. Add the line

os : `os`

to your configuration (say, test.cfg).

2. Access it like this:

from config import Config
cfg = Config(file('test.cfg'))
cfg.os.environ['OS']

'Windows_NT'

The point, of course, is that you can refer to environment variables
from elsewhere in your configuration, e.g.

temp_path: `os.environ['TEMP']`
work_file_prefix: $temp_path + `os.sep`

Best regards,


Vinay Sajip

 

[Vinay Sajip]

I know you probably want a lot of features, but then different people
want different things. What a person considers necessary, others may
find them unnecessary, and vice versa.

I'm not sure what you mean by 'lots of features'. Primarily, I wanted
better ease of use than ConfigParser gives me for medium-complexity
configurations.

Microsoft went from .INI files to Windows registry, and now to
XML-based app.config files in the .NET world. I know app.config files
are not the best thing with the cleanest format, but, why re-invent
the wheel?

Microsoft did. You may remember, when the registry first appeared in
Windows, many viewed it as some kind of panacea - it's the same thing
now with .NET configuration files. Do you really believe that they are
the last word in application configuration? I don't. Of course, they are
quite usable, and have a lot of momentum since anything done by
Microsoft does, but that's hardly a reason for people to stop trying
alternative approaches.

Microsoft's XML-based app.config is guaranteed to be widely used. For
something as minor as configuration files, for people that juggle
between many programming languages, I guess it's an OK solution.
I know many people may not like the XML overhead, I know many people
don't like Microsoft. But when you have to use hierarchical
configuration files across many programming languages, I don't know
any format out there that can compete with app.config. To me, the war
is already over.

Which war would that be? Horses for courses, of course. XML is
technically suitable, and the .NET System.Configuration format will seem
like a de facto standard for many people. I've used it myself on many a
..NET project. (Normally, end users are not expected to edit these
configuration files, as they are XML tyros and moreover, cannot be
expected to understand the System.Configuration schema.)

If you'd rather use the .NET format for your Python application
configuration, fine!

Best wishes,


Vinay Sajip

 

[Sylvain Thenault]

I'm interested to see how your bridging works.

well, the basic idea is a class providing a (simple) unified api to
register options and to read them from different sources, those
methods taking care of optparse / ConfigParser particularities

A basic example showing main functionalities :

---- begin example

import sys
from logilab.common.configuration import Configuration

options = [('dothis', {'type':'yn', 'default': True, 'metavar': '<y or n>'}),

.... ('value', {'type': 'string', 'metavar': '<string>'}),
.... ('multiple', {'type': 'csv', 'default': ('yop',),
.... 'metavar': '<comma separated values>',
.... 'help': 'you can also document the option'}),

config = Configuration(options=options, name='My config')
print config['dothis'] True
print config['value'] None
print config['multiple'] ('yop',)
print config['number'] 2

config.help()

usage: [options]

options:
-h, --help show this help message and exit

My config:
--dothis=<y or n>
--value=<string>
--multiple=<comma separated values>
you can also document the option

f = open('myconfig.ini', 'w')
f.write('''[MY CONFIG]

.... number = 3
.... dothis = no
.... multiple = 1,2,3
.... ''')

f.close()
config.load_file_configuration('myconfig.ini')
print config['dothis'] 0
print config['value'] None
print config['multiple'] ['1', '2', '3']
print config['number'] 3

sys.argv = ['mon prog', '--value', 'bacon', '--multiple', '4,5,6', 'nonoptionargument']
print config.load_command_line_configuration() ['nonoptionargument']
print config['value'] bacon

config.generate_config()

[MY CONFIG]
dothis=no

value='bacon'

# you can also document the option
multiple=4,5,6

number=3

---- end example

Using the config module I released, access to environment variables is
very easy. Here's how:

1. Add the line

os : `os`

to your configuration (say, test.cfg).

I think the idea was to ask in the configuration file to specify a value
should be taken from an environment variable, but to get a value from an
environment variable and / or from the configuration file.

2. Access it like this:

from config import Config
cfg = Config(file('test.cfg'))
cfg.os.environ['OS']

'Windows_NT'

The point, of course, is that you can refer to environment variables
from elsewhere in your configuration, e.g.

temp_path: `os.environ['TEMP']`
work_file_prefix: $temp_path + `os.sep`

I've one problem with this approach (other that the person writing the
configuration file has to know python to benefit from your module). Is
there something preventing things like :

work_file_prefix: `os.remove('/etc/password')`

 

[Vinay Sajip]

well, the basic idea is a class providing a (simple) unified api to
register options and to read them from different sources, those
methods taking care of optparse / ConfigParser particularities

[example snipped]

Thanks for sharing this.

I think the idea was to ask in the configuration file to specify a value
should be taken from an environment variable, but to get a value from an
environment variable and / or from the configuration file.

I think the config module is flexible enough to allow a number of
different approaches to be used, including the one you mention. It's
flexible enough to allow configuration values to be looked for in a
number of places: see

http://www.red-dove.com/python_config.html#cascading-configurations

for an example. One of the cascaded configurations could provide access
to options provided via the command line, as per my earlier post.

I've one problem with this approach (other that the person writing the
configuration file has to know python to benefit from your module). Is
there something preventing things like :

work_file_prefix: `os.remove('/etc/password')`

Configuration files read by the config module are *not* Python, though
the dict and list syntax is very Python-like. Hence the syntax
`os.remove('/etc/password')` would give a syntax error during parsing.
The only forms allowed are an identifier followed by zero or more
suffixes, which are one of:

..identifier
[number]
['string'] (or ["string"])

So you can say `os.sep` and `os.environ['TEMP']` but not call any
methods explicitly. The back-tick may appear misleading because it may
imply unrestricted execution - if you can suggest a better character to
use, please do!

 

[Steven Bethard]

A basic example showing main functionalities :

[snip very cool example]

Wow, this is great! This is exactly the kind of thing I was looking for the
last time I had to write config files...

Steve

 

[Reinhold Birkenfeld]

It's just an idea and I haven't put a lot of thought in it myself, but what
about using XML for the configuration syntax?

I'm currently using YAML for configuration syntax. I don't need object
references as in the OP's module, so I'm happy with it.

It's also easy to change for users.

Reinhold

 

[Diez B. Roggisch]

I'm currently using YAML for configuration syntax. I don't need object

references as in the OP's module, so I'm happy with it.

afaik yauml supports these, too. And it even carries type information, which
can be nice but sometimes sucks if you want your config file invariant of
class names or actual layout.

It's also easy to change for users.

Yes, and there exist bindings for different languages. It also tries to be
clever about data types.

 

[Dan Perl]

Thanks for the suggestion. What are you using for a python implementation?
I saw that there is PyYAML and Syck (multi-language, including python).

Dan

It's just an idea and I haven't put a lot of thought in it myself, but
what
about using XML for the configuration syntax?

I'm currently using YAML for configuration syntax. I don't need object
references as in the OP's module, so I'm happy with it.

It's also easy to change for users.

Reinhold


--
[Windows ist wie] die Bahn: Man muss sich um nichts kuemmern, zahlt fuer
jede Kleinigkeit einen Aufpreis, der Service ist mies, Fremde koennen
jederzeit einsteigen, es ist unflexibel und zu allen anderen Verkehrs-
mitteln inkompatibel. -- Florian Diesch in dcoulm