initializing mutable class attributes

[Alex Martelli]

I found a small hole in the initial code suggestion and I fixed it. Try and
find the difference:
class test(object):
def __new__(typ, *args, **kwargs):
obj = object.__new__(typ)
obj.attr1 = 666
typ.__init__(obj, *args, **kwargs)
return obj

class derived(test):
def __init__(self, arg1, arg2):
self.attr2 = arg1
self.attr3 = arg2

d = derived(111, 222)
print d.attr1, d.attr2
print isinstance(d, test)

So, what do people think about it? Alex, would this be good for a recipe in
the Python Cookbook?

I might run it side by side with my custom metaclass, sure. I'm not
sure why you're running typ.__init__ from within test.__new__ -- it
seems to me that will result in double-running cases.


Alex

 

[Alex Martelli]

Here's a way to auto-initialize an instance attribute to a mutable via a class variable
(that happens to be a descriptor ;-)
... class prop(object):
... def __get__(self, inst, cls=None):
... if inst: return inst.__dict__.setdefault('prop',[])
... return self
... def __set__(self, inst, value): inst.__dict__['prop'] = value
... def __delete__(self, inst): del inst.__dict__['prop']
... prop = prop()
...

I think it's the most elegant approach so far, except I would further
tweak it as follows:

class auto_initialized(object):
def __init__(self, name, callable, *args, **kwds):
self.data = name, callable, args, kwds
def __get__(self, ints, cls=None):
if inst:
name, callable, args, kwds = self.data
self = inst.__dict__[name] = callable(*args, **kwds)
return self

No __set__ and __del__ warranted, I believe. Use would then be, e.g:

class C(object):
prop = auto_initialized('prop', list)

Pity that the name must be specified, but ah well, that's descriptors'
defects -- they don't get reminded about the name upon __get__ etc.

Any use to you? (disclaimer: not tested beyond what you see here ;-)

I could use it in the Cookbook, side by side with Dan's "overloading
__new__" idea and my custom metaclass, as various approaches to solving
a 'problem' (if that be one) of "automatically initializing mutable
instance attributes". It would help if you and/or Dan posted these
recipes to ActiveState's online cookbook ASAP (deadline is Sat Sep 5...)
-- I'm allowed to add recipes that didn't show up on the online
cookbook, but that _is_ discouraged... (and posting the recipes myself
isn't much better, either).


Alex

 

[Alex Martelli]

(...)

I do find it reasonably common (particularly in GUI applications when
I'm subclassing standard GUI objects) to have a subclass that massages
the constructor inputs slightly (perhaps adding different defaults, or
filtering the arguments) before passing them on to the superclass.

In C++ you can sometimes manage this as long as the manipulation is
simple exclusion or addition that can be performed in the
initialization code prior to the start of the subclass constructor,
but that's very limited (and otherwise can end up forcing you into the
2-phase construction model). So I find Python's freedom of
controlling exactly when the superclass is initialized with complete
control over its arguments very liberating.

Very good explanation of the key reason why in C++ you end up using the
pattern "two phase constructor" far more often than in Python (although
it's not _foreign_ to Python, mind you)...


Alex

 

[David Bolen]

That is exactly what I'm trying to do, and in the initial posting of the
thread I was mentioning an initialization exactly like yours, but at the
same time I was pointing out that an initialization like that works only for
immutable attributes and not for mutable attributes, hence the need for the
__init__. Or doing it in __new__ as I showed in the new code.

Oh (glancing around with an embarrassed look) - you mean the behavior
that's the topic of this thread (generally easily determined by
reading the subject line to which one is replying)...



-- David

 

[Dan Perl]

...
I could use it in the Cookbook, side by side with Dan's "overloading
__new__" idea and my custom metaclass, as various approaches to solving
a 'problem' (if that be one) of "automatically initializing mutable
instance attributes". It would help if you and/or Dan posted these
recipes to ActiveState's online cookbook ASAP (deadline is Sat Sep 5...)
-- I'm allowed to add recipes that didn't show up on the online
cookbook, but that _is_ discouraged... (and posting the recipes myself
isn't much better, either).

I already submitted mine. It's recipe #303059, so go to:
http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/303059

You are right in an earlier posting that another call to __init__ in the
__new__ was redundant. It was even a problem when using it in the multiple
inheritance case. I figured out eventually that it's not needed and I got
rid of it.

Dan

 

[Dan Perl]

........

Incidentally, FAR from your assertion that, in C++, "subclasses don't
need to know anything about how the superclass is implemented", reality
is just the other way 'round. Subclasses have to be _intimately
familiar_ with key aspects of superclasses' implementation -- not just
the interface (which fully includes the "little detail" of whether a
class has a default ctor or not!!!), but specifically _implementation_
issues, such as what private member names the superclass uses for its
own implementation purposes. That's because C++ fatally tries to
support encapsulation via _accessibility_, NOT through _visibility_. If
the subclass unwittingly tries using a member name which the superclass
is already using privately, it just won't compile; if the superclass
adds any private names to its implementation from one release to
another, it will "break all user code" which quite correctly tried to
use that name (and previously managed w/o problems!) for one of its
members -- as you say.

No, Alex. Data members in a C++ subclass cannot conflict with data members
in its superclass, public, protected, or private. They are resolved by
their namespaces. See an example:
#include <iostream>

class Class1 {
public:
void setM1(int arg) {m1=arg;}
void getM1( ) {std::cout << "c1m1: " << m1 << std::endl;}
private:
int m1;
};
class Class2 : public Class1 {
public:
void setM1(int c1m1, int c2m1) {Class1::setM1(c1m1); m1=c2m1;}
void getM1( ) {Class1::getM1( ); std::cout << "c2m1: " << m1 <<
std::endl;}
private:
int m1;
};
int main(int argc, char **argv)
{
Class2 c2 = Class2( );
c2.setM1(1, 9);
c2.getM1( );
}

Here it is -- now feel free to damage yourself at will with your
conviction that this (calling superclass ctors) "IS a case" where
implicit is better. I decided I can reuse this as a Cookbook recipe,
after all!-)

I'll look at the metaclass example later. Thanks.

Dan

 

[Shalabh Chaturvedi]

Someone else (Shalabh) suggested descriptors for the same problem but I
didn't get to consider such a solution until now.

This is what I had in mind:

---->8-----
class defaultvalue(object): # this is a descriptor
def __init__(self, name, factory):
self.name = name
self.factory = factory

def __get__(self, obj, cls=None):
if obj is None:
return self

val = self.factory()
setattr(obj, self.name, val)
return val


class C(object):
i = defaultvalue('i',dict)
j = defaultvalue('j',list)

c = C()
print c.i, c.j # prints {} []
---->8-----

Once it has kicked in, it's free of the descriptor overhead. Note you
only need to define defaultvalue once and reuse it everywhere. Also you
can give it a function or lambda like:

k = defaultvalue('k', lambda :[1,2,3])

I still suggest you be a good Pythoneer and use __init__() like everyone
else. It's a useful habit of always calling the base class __init__() at
the top of your __init__(). If you don't develop this habit (or of at
least checking the base for __init__), you will waste debugging cycles
when you use other libraries.

 

[Dan Perl]

I really like this one. I think this 'defaultvalue' class even deserves to
be added as a builtin. It can even be changed to initialize non-empty
sequences. And I haven't tried that yet, but it should be able to
initialize an attribute with any kind of object that can be constructed with
no arguments. Too bad it works only for instance attributes in a class.

I will be a good Pythoneer/Pythonista and I will invoke parent __init__'s
whenever using a library. My concern was not as a library user, but as a
library developer. In which case I don't like the idea of relying on users
to be good Python coders.

Dan

Someone else (Shalabh) suggested descriptors for the same problem but I
didn't get to consider such a solution until now.

This is what I had in mind:

---->8-----
class defaultvalue(object): # this is a descriptor
def __init__(self, name, factory):
self.name = name
self.factory = factory

def __get__(self, obj, cls=None):
if obj is None:
return self

val = self.factory()
setattr(obj, self.name, val)
return val


class C(object):
i = defaultvalue('i',dict)
j = defaultvalue('j',list)

c = C()
print c.i, c.j # prints {} []
---->8-----

Once it has kicked in, it's free of the descriptor overhead. Note you
only need to define defaultvalue once and reuse it everywhere. Also you
can give it a function or lambda like:

k = defaultvalue('k', lambda :[1,2,3])

I still suggest you be a good Pythoneer and use __init__() like everyone
else. It's a useful habit of always calling the base class __init__() at
the top of your __init__(). If you don't develop this habit (or of at
least checking the base for __init__), you will waste debugging cycles
when you use other libraries.

 

[Shalabh Chaturvedi]

I will be a good Pythoneer/Pythonista and I will invoke parent __init__'s
whenever using a library. My concern was not as a library user, but as a
library developer. In which case I don't like the idea of relying on users
to be good Python coders.

Don't worry about that - just send them to c.l.py and here we'll badger
them till they succumb and promise to call their parent __init__s
properly <wink>.

Seriously though, I think it's part of Python ideology to trust the
programmer a little more. Notice no private or protected members.
Keeping things explicit and not implicit such as in this case trains
users very quickly. They'll remember to call the parent __init__, or
figure out very quickly when it doesn't work (hopefully with useful
error messages). If things were happening behind the scenes, users will
have a hard time finding out what's going on when it does not work as
expected. Or if they are power users and want to tweak something,
they'll may have to burrow around quite a bit.

That said, in cases it may actually be useful to do certain things
automatically. After all, making life easier for the user is certainly a
good thing. IMO, it's important to accomodate needs of all kinds of
users, but not their fancies. From what I've seen, that seems to be the
way Python itself is developed.

Just my $0.02.

Cheers,
Shalabh

 

[Dan Perl]

.........

Seriously though, I think it's part of Python ideology to trust the
programmer a little more. Notice no private or protected members.
Keeping things explicit and not implicit such as in this case trains
users very quickly. They'll remember to call the parent __init__, or
figure out very quickly when it doesn't work (hopefully with useful
error messages). If things were happening behind the scenes, users will
have a hard time finding out what's going on when it does not work as
expected. Or if they are power users and want to tweak something,
they'll may have to burrow around quite a bit.

That said, in cases it may actually be useful to do certain things
automatically. After all, making life easier for the user is certainly a
good thing. IMO, it's important to accomodate needs of all kinds of
users, but not their fancies. From what I've seen, that seems to be the
way Python itself is developed.

I think that what you're saying and what other people have been saying is
that this is an idiom and it's better to follow idioms in a language because
it helps people understand your code. I agree with that and I guess I will
end up following this idiom. In the end, it's a choice between coding for
the beginners or coding for the experienced. In general, that's not an
obvious choice, but this seems to be quite a common idiom, so the ratio of
beginners who don't know it to people who do know it should be small.

Dan

 

[Alex Martelli]

........ ...
No, Alex. Data members in a C++ subclass cannot conflict with data members
in its superclass, public, protected, or private. They are resolved by

Sorry, I didn't explain clearly the breakage example -- try this one,
which is as tiny as I can make it:

#include "Base.h"

int foo = 23;
class Derived: public Base {
public:
Derived() { foo = 45; }
};

Is this code correct? Sure, it compiles and runs perfectly using Base
version 12.3. Then Base comes out with version 12.4 and this does not
compile any more. Why? Because the new version of Base has added a
private member named foo.

Admittedly it breaks at compile-time, which is a bit better than
breaking at unittest-time as it would if Base had grown a _public_
member named foo (as the global foo doesn't get updated any more but
rather the per-instance one does). But the point is: coupling between
superclasses and subclasses is STRONG. To code a subclass, you can't
ignore what the superclass is doing, including some implementation
aspects -- much less of course ignore aspects of the superclass's public
interface, as you erroneously claimed.


Alex

 

[Alex Martelli]

I really like this one. I think this 'defaultvalue' class even deserves to
be added as a builtin. It can even be changed to initialize non-empty
sequences. And I haven't tried that yet, but it should be able to
initialize an attribute with any kind of object that can be constructed with
no arguments. Too bad it works only for instance attributes in a class.

As I posted on the other subthread, you can perfectly well use arguments
in the call, too -- just add *args and **kwds to the descriptor's
__init__ and use them when you later call the factory. This is an
important patter in Python: whenever you design a callable X which
accepts a callable Y as an argument, consider adding *args and **kwds to
X's signature, stashing them away together with Y, and passing them back
into Y when the time comes to call it. This can save users of X writing
lambdas or def functions which have no reason to exist except fixing
such parameters.

I will be a good Pythoneer/Pythonista and I will invoke parent __init__'s
whenever using a library. My concern was not as a library user, but as a
library developer. In which case I don't like the idea of relying on users
to be good Python coders.

If they don't follow good, normal coding practices there is really
little you can do to save them from themselves. Surely nothing worth
distorting all OO design to ensure every class can be meaningfully
initialized without parameters, as I explained in another part of this
thread.


Alex

 

[Andrew Dalke]

Alex:

#include "Base.h"

int foo = 23;
class Derived: public Base {
public:
Derived() { foo = 45; }
};

Is this code correct? Sure, it compiles and runs perfectly using Base
version 12.3. Then Base comes out with version 12.4 and this does not
compile any more. Why? Because the new version of Base has added a
private member named foo.

My C++ skills are rusty and old. I had to experiment
with this to understand what was going on. I had thought
that private: kept the above from happening. Now I know
better.

I was wondering though about the justification of
"is this code correct" using "compiles and runs perfectly"
under a given circumstance. That justification has
been applied to many incorrect pieces of code, including

#define SQUARE(x) ((x)*(x))

only to be proven wrong in the real world. Part
of learning the language is learning about these
gotchas and acquiring the skills needed to do it
correctly.


The problem in your example is that variable lookup
occurs first in class then in static scopes. I can
tell the compiler to only use static scope, as in

Derived() { ::foo = 45; }

Based on my creaky memories, this shouldn't have
the same problems. Does it have others? And is
this this proper and accepted solution, meaning
that I should always write an explicit static
scope when I want to ensure I don't accidently
conflict with future changes to the base class?

Andrew
(e-mail address removed)

P.S.
Congratulations to the both of you!


P.P.S.

Is the modern implementation of that DOUBLE macro
something like

inline template<T>
T& DOUBLE(const T& x) {
return T*T;
}

? I don't think so since it doesn't do coercion
the same way. Consider
float f = .... ;
double x = SQUARE(f);

The f*f gets promoted to double. The macro version
assigns the double to a double so keeps the precision.
OTOH, the template version narrows the double to
a float (for the return value) then widens that
float back to a double, hence loosing precision in
the process.