[rfc] An object that creates (nested) attributes automatically onassignment

[Edd]

Hi folks,

I'd like to use Python itself as the configuration language for my
Python application. I'd like the user to be able to write something
like this in their config file(s):

cfg.laser.on = True
cfg.laser.colour = 'blue'
cfg.discombobulated.vegetables = ['carrots', 'broccoli']
# ...

To this end, I've created a class that appears to allow instance
variables to be created on the fly. In other words, I can to the
following to read a config file:

cfg = Config()
execfile(filename, {'cfg', cfg}, {})

However, I think my implementation of the Config class is a little
crappy. I'd really appreciate the critical eye of a pro. Here's the
sauce:

class Config(object):
def __init__(self, sealed=False):
def seal():
for v in self._attribs.values():
if isinstance(v, self.__class__): v.seal()
del self.__dict__['seal']

d = {'_attribs': {}, '_a2p': None}
if not sealed: d['seal'] = seal

self.__dict__.update(d)

def __getattr__(self, key):
if not key in self._attribs:
d = Config(sealed='seal' not in self.__dict__)
def add2parent():
self._attribs[key] = d
if self._a2p:
self._a2p()
self._a2p = None

# if anything is assigned to an attribute of d,
# make sure that d is recorded as an attribute of this
Config
d._a2p = add2parent
return d
else:
return self._attribs[key]

def __setattr__(self, key, value):
if key in self.__dict__:
self.__dict__[key] = value
else:
if not 'seal' in self.__dict__:
clsname = self.__class__.__name__
raise AttributeError("'%s' object attribute '%s'
is read-only (object is sealed)" % (clsname, key))
self.__dict__['_attribs'][key] = value
if self._a2p:
self._a2p()
self._a2p = None

def __delattr__(self, key):
if key in self.__dict__:
clsname = self.__class__.__name__
raise AttributeError("can't delete '%s' object
attribute '%s' as it is used for book-keeping!" % (clsname, key))
else:
if key in self._attribs:
del self._attribs[key]

def __bool__(self):
return bool(self._attribs)

def __nonzero__(self):
return bool(self._attribs)

if __name__ == '__main__':
cfg = Config()
cfg.a = 1
cfg.b.c = 2
cfg.d.e.f.g.h = [1, 2, 3]
print cfg.a
print cfg.b.c
print cfg.d.e.f.g.h

del cfg.b.c
print cfg.b.c

try:
del cfg.d.e._attribs
except AttributeError, ex:
print ex

cfg.seal()
try:
cfg.k.l.z = []
except AttributeError, ex:
print ex

Once the config is loaded, it will be passed down to other user-
written scripts and it's important that these scripts don't
accidentally change the config. So the idea is that I'll call cfg.seal
() to prevent any further changes before passing it on to these other
scripts. Beyond the general fiddliness of the code, I think the way
seal() currently works is particularly pants.

I considered using a simpler approach:

def mkdd(): return defaultdict(mkdd)
cfg = mkdd()
execfile(filename, {'cfg': cfg}, {})

But I quite like the way the '.' separators quite naturally (IMO)
indicate a hierarchy of settings.

Comments and suggestions welcome!

Kind regards,

Edd

 

[Steven D'Aprano]

Hi folks,

I'd like to use Python itself as the configuration language for my
Python application. I'd like the user to be able to write something like
this in their config file(s):

cfg.laser.on = True
cfg.laser.colour = 'blue'
cfg.discombobulated.vegetables = ['carrots', 'broccoli'] # ...

To this end, I've created a class that appears to allow instance
variables to be created on the fly.

Um, don't all classes allow that?

Or do you mean instance *attributes*? Again, apart from built-in types
and classes that use __slots__, all classes allow that.

In other words, I can to the
following to read a config file:

cfg = Config()
execfile(filename, {'cfg', cfg}, {})

That's okay so long as you trust the user not to put malicious, or buggy,
code in your config file. Personally, I think config files should be more
tolerant of errors than a programming language.

However, I think my implementation of the Config class is a little
crappy. I'd really appreciate the critical eye of a pro. Here's the
sauce:

For starters, where is your documentation? No doc strings, not even any
comments! No, I tell a lie... *one* obscure comment that doesn't really
explain much.

class Config(object):
def __init__(self, sealed=False):
def seal():
for v in self._attribs.values():
if isinstance(v, self.__class__): v.seal()
del self.__dict__['seal']

d = {'_attribs': {}, '_a2p': None}
if not sealed: d['seal'] = seal

self.__dict__.update(d)

I'm going to try to guess what the above does. When you initialise an
instance, you can tell the instance to be "sealed" or unsealed. I'm not
sure what the difference is, or why you would choose one over the other.
Sealed instances seem to be exactly the same as unsealed instances,
except they have a seal() method (actually a closure). The seal method,
when called, recursively seals any embedded Config instances inside the
current instance, then deletes itself.

Arghhh!!! Self-modifying code!!! Unclean, unclean!!!

I'm not sure why seal() is necessary -- it seems to me that if present,
all it does is delete itself. So why not just leave it out altogether?

You also have a rather complicated way of adding instance attributes.
Instead of

d = {'_attribs': {}, '_a2p': None}
self.__dict__.update(d)

why not just do the more obvious:

self._attribs = {}
self._a2p = None

?

def __getattr__(self, key):
if not key in self._attribs:
d = Config(sealed='seal' not in self.__dict__) def
add2parent():
self._attribs[key] = d
if self._a2p:
self._a2p()
self._a2p = None

It looks like you are just re-inventing the normal attribute mechanism of
Python. I'm not sure why you feel this is necessary. And it contains MORE
self-modifying code! Yuck! Frankly I don't care enough to dig into your
code to understand how it works in detail.

# if anything is assigned to an attribute of d, # make
sure that d is recorded as an attribute of this
Config
d._a2p = add2parent
return d
else:
return self._attribs[key]

def __setattr__(self, key, value):
if key in self.__dict__:
self.__dict__[key] = value
else:
if not 'seal' in self.__dict__:
clsname = self.__class__.__name__
raise AttributeError("'%s' object attribute '%s'
is read-only (object is sealed)" % (clsname, key))
self.__dict__['_attribs'][key] = value if self._a2p:
self._a2p()
self._a2p = None

Does "sealed" mean that the instance is read-only? If so, and if I'm
reading this correctly, I think it is buggy. You allow modifications to
attributes inside __dict__ *without* checking to see if the instance is
read-only. Then you get the test backwards: surely the existence, not the
absence, of a 'seal' attribute should mean it is sealed?

def __delattr__(self, key):
if key in self.__dict__:
clsname = self.__class__.__name__
raise AttributeError("can't delete '%s' object
attribute '%s' as it is used for book-keeping!" % (clsname, key))
else:
if key in self._attribs:
del self._attribs[key]

Nothing much to say here, except that you're doing more work re-inventing
the wheel, storing attributes inside _attribs instead of using the
general attribute mechanism. Seems unnecessary to me, but perhaps I don't
understand your use-case.

Once the config is loaded, it will be passed down to other user- written
scripts and it's important that these scripts don't accidentally change
the config. So the idea is that I'll call cfg.seal () to prevent any
further changes before passing it on to these other scripts.

Or you could pass a *copy* of the config, and let them change it to their
heart's content, it won't matter.

Or you could say "we're all adults here", simply document that any
changes will have consequences, and let user scripts change the config.
And why not?

Beyond the
general fiddliness of the code, I think the way seal() currently works
is particularly pants.

Is "pants" slang for "fragile, hard to understand and difficult to debug"?

 

[John Machin]

Is "pants" slang for "fragile, hard to understand and difficult to debug"?

pommy slang for "sucks intensely, like the Deathstar's tractor
beam" ... I think we agree with him.

 

[Edd]

Hi Steven,

Thank you for your response!

Hi folks,

I'd like to use Python itself as the configuration language for my
Python application. I'd like the user to be able to write something like
this in their config file(s):

   cfg.laser.on = True
   cfg.laser.colour = 'blue'
   cfg.discombobulated.vegetables = ['carrots', 'broccoli'] # ...

To this end, I've created a class that appears to allow instance
variables to be created on the fly.

Um, don't all classes allow that?

Or do you mean instance *attributes*? Again, apart from built-in types
and classes that use __slots__, all classes allow that.

Yes I probably mean instance attributes. Forgive me, I am not
particularly sure of the terminology. But your MyClass example, won't
quite do what I want, as I'd like to be able to define instance
attributes on top of instance attributes by assignment:
Traceback (most recent call last):

That's okay so long as you trust the user not to put malicious, or buggy,
code in your config file. Personally, I think config files should be more
tolerant of errors than a programming language.

That's certainly a valid remark, but this will be a tool for
programmers. I am hoping that the user will make use of the power in
moderation. Often, it really will be useful to allow functions to be
defined in the config files, for example.

For starters, where is your documentation? No doc strings, not even any
comments! No, I tell a lie... *one* obscure comment that doesn't really
explain much.

Yes, you're quite right. I was about to add some doc strings, but I
didn't think the implementation was good enough. That's somewhat
backwards, though, right?! Especially considering I'm asking for
improvements. Anyway, I had hoped that the example usage at the end
would show what the purpose of the class is.

    class Config(object):
        def __init__(self, sealed=False):
            def seal():
                for v in self._attribs.values():
                    if isinstance(v, self.__class__): v.seal()
                del self.__dict__['seal']

            d =  {'_attribs': {}, '_a2p': None}
            if not sealed: d['seal'] = seal

            self.__dict__.update(d)

I'm going to try to guess what the above does. When you initialise an
instance, you can tell the instance to be "sealed" or unsealed. I'm not
sure what the difference is, or why you would choose one over the other.
Sealed instances seem to be exactly the same as unsealed instances,
except they have a seal() method (actually a closure). The seal method,
when called, recursively seals any embedded Config instances inside the
current instance, then deletes itself.

Arghhh!!! Self-modifying code!!! Unclean, unclean!!!
Quite!

I'm not sure why seal() is necessary -- it seems to me that if present,
all it does is delete itself. So why not just leave it out altogether?

As I said in the original post, such Config objects will be made
available to other kinds of user-written script and it's important
that the Config not change between the execution of one script and the
next. The seal() mechanism was an attempt to help the user from
*accidentally* doing this and then having to try to diagnose the
problem and understand how changing the config might have broken the
invariants of the software. I guess a big "DON'T CHANGE THE CONFIG IN
YOUR SCRIPTS" message in the manual, might be sufficient, though

You also have a rather complicated way of adding instance attributes.
Instead of

d =  {'_attribs': {}, '_a2p': None}
self.__dict__.update(d)

why not just do the more obvious:

self._attribs = {}
self._a2p = None

Because that would go through __setattr__(), which does something else
(which is the whole point of the class). At least, that was my
understanding, which certainly could be at fault.

This might be nicer I guess:

self.__dict__['_attribs'] = {}
self.__dict__['_a2p'] = None

There used to be more instance attributes than just two so it was
easier to put them in a dict and use update. I agree that it's rather
obfuscated, though.

[Edd's horrendous code snipped]

It looks like you are just re-inventing the normal attribute mechanism of
Python. I'm not sure why you feel this is necessary. And it contains MORE
self-modifying code! Yuck! Frankly I don't care enough to dig into your
code to understand how it works in detail.

Ok. But is there a quick-and-easy way of creating an object, cfg, such
that I can write:

cfg.hovercraft.full.of = 'eels'

without knowing in advance that the user will want a .hovercraft
instance attribute, or a .full attribute inside that, or a .of inside
that, or ... ?

        def __setattr__(self, key, value):
            if key in self.__dict__:
                self.__dict__[key] = value
            else:
                if not 'seal' in self.__dict__:
                    clsname = self.__class__.__name__
                    raise AttributeError("'%s' object attribute '%s'
is read-only (object is sealed)" % (clsname, key))
                self.__dict__['_attribs'][key] = value if self._a2p:
                    self._a2p()
                    self._a2p = None

Does "sealed" mean that the instance is read-only? If so, and if I'm
reading this correctly, I think it is buggy. You allow modifications to
attributes inside __dict__ *without* checking to see if the instance is
read-only. Then you get the test backwards: surely the existence, not the
absence, of a 'seal' attribute should mean it is sealed?

The absence of the seal() 'method' means that it has already been
called i.e. the object has already been sealed. I agree it's somewhat
fishy, which is why I'm asking for suggestions for improvements.
Perhaps I should just forget the seal() idea.

Or you could pass a *copy* of the config, and let them change it to their
heart's content, it won't matter.

I think that's probably the best thing. I was worried about some parts
not being deep-copyable, but I think I'm happy to put that concern
aside. Besides, right now, even though you can't add/delete attributes
in a Config, you can still change existing ones:

cfg = Config()
cfg.a.b.c = [1, 2]
cfg.seal()
cfg.a.b.c[1] = 3 # pffff

Yes, you've helped convince me that it's just a bad idea.

Or you could say "we're all adults here", simply document that any
changes will have consequences, and let user scripts change the config.
And why not?

Even adults make the occasional tiny mistake which has confusing
consequences in a larger system. It was an attempt to help prevent
this. I probably worry too much about that.

Is "pants" slang for "fragile, hard to understand and difficult to debug"?

Yes! Rest assured that I am under no illusion that what I have written
is good!

Steven, I greatly appreciate your taking the time to understand the
aforementioned horrors. Assuming that the seal() stuff is no longer a
requirement, is there a cleaner way of creating an object where
(nested) instance attributes can be defined by simple assignment?

Perhaps it would have been better if I had left out my awful attempt
altogether. It seems like it only made my question more confusing than
it needed to be If it would help, I'd be happy to add some doc
strings and tests but I was getting ready to throw this code away when
a cleaner 5-line alternative was presented!

Kind regards,

Edd

 

[Steven D'Aprano]

Yes I probably mean instance attributes. Forgive me, I am not
particularly sure of the terminology. But your MyClass example, won't
quite do what I want, as I'd like to be able to define instance
attributes on top of instance attributes by assignment:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: MyClass instance has no attribute 'laser'

Ah, now it is more clear.

Okay, let's try this:

.... def __getattr__(self, name):
.... # Only called if self.name doesn't exist.
.... inst = self.__class__()
.... setattr(self, name, inst)
.... return inst
....45

 

[Edd]

Ah, now it is more clear.

Okay, let's try this:


...     def __getattr__(self, name):
...             # Only called if self.name doesn't exist.
...             inst = self.__class__()
...             setattr(self, name, inst)
...             return inst

Ha! Perfect! I knew it should be simpler. Thanks very much!

Kind regards,

Edd