lambda

[Steve Holden]

Antoon Pardon wrote:

Op 2005-01-17, Steve Holden schreef <[email protected]>:


Antoon Pardon wrote:
[...]


"A foolish consistency is the hobgoblin of little minds". Rules are made
to be broken.


Like only use immutables as dictionary keys.


Fair enough, but don;t go advising newbies to do this.


How about something like this.

Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

There you go with the minutiae again. How about:

"Don't use mutables as hash keys"?

That sounds too dogmatic to my ears. I also find it
too selective. The problem with mutables as dictionary
keys is not specific to dictionaries. Everywhere you
have mutables in a container, it is possible that
mutating the object in the container will cause
problem. Heck even using mutables as arguments can
cause trouble. Why else the specific advice against

def foo(p = [])

type of arguments. So should we adopt the principles:

Don't use mutables in containers

Don't use mutables as default values for parameters

Don't use mutables as arguments.

Don't assign one mutable to an other.


I don't see a big difference between these principles
and the hash key principle, so in the end may be we
should just stick with the more general principle:

Don't use mutables!


and be done with it.

http://redwing.hutman.net/~mreed/warriorshtm/tirelessrebutter.htm

regards
Steve

 

[Antoon Pardon]

The main difference is: if you mutate a dict key you *always* have a
problem. So don't do that. Mutating (say) a list item *only* is a
problem if you (say) need that list to remain sorted.

That is not true. It is a problem every time I expect the list
items to remain the same.

Lists don't
magically remain sorted, so people generally sort it before they do some
operation that expects a sorted list.

Heck even using mutables as arguments can
cause trouble. Why else the specific advice against

def foo(p = [])

type of arguments. So should we adopt the principles:

Don't use mutables in containers
Nonsense.

Don't use mutables as default values for parameters

That's good advice in general.

Don't use mutables as arguments.
Nonsense.

Don't assign one mutable to an other.

Nonsense. Some newbies get surprised by Python's assignment-doesn't-copy
semantics, but it's such basic knowledge (as well as a useful feature)
that I simply don't understand you saying this.

Well it is this same sematics that causes the problems with
mutable dictionary keys. If it is such basic knowledge then what
is the problem with mutable dictionary keys?

Than you haven't looked hard enough.

All of these can get unexpected behaviour because of the
assignment-doesn't-copy semantics. The same semantics
that can cause problems if you work with mutable dictionary
keys.

Ok, so you're indeed a troll.

The problems with mutables as dictionary keys is just one
particulary case of the problems you can have when your
assignmentxs semantics just creates a new reference to the
same object. As such it is no different from the problem
of mutating one object and finding out this object was
also reference through an other name you expected to remain
the same or finding out this object was also in a list you
expected to remain stable etc.

 

[Antoon Pardon]

http://redwing.hutman.net/~mreed/warriorshtm/tirelessrebutter.htm

regards
Steve

I you want to play it like this

http://redwing.hutman.net/~mreed/warriorshtm/crybaby.htm

 

[John Lenton]

Antoon Pardon wrote:
[...]

"A foolish consistency is the hobgoblin of little minds". Rules are made
to be broken.


Like only use immutables as dictionary keys.

Fair enough, but don;t go advising newbies to do this.

How about something like this.

Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

knowledgeable and experienced users know when to ignore the rules.

--
John Lenton ([email protected]) -- Random fortune:
Una buena gran parte del arte del bien hablar consiste en saber mentir con
gracia.
-- Erasmo de Rotterdam. (1469-1536).

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFB69asgPqu395ykGsRAkNNAJwPy4YOsAUoUHWTJc6M1vWqdgquIwCdFCaf
LokGmkEUZL/0ezwbzMpRScc=
=XKjw
-----END PGP SIGNATURE-----

 

[Antoon Pardon]

--vni90+aGYgRvsTuO
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Antoon Pardon wrote:
[...]
"A foolish consistency is the hobgoblin of little minds". Rules are ma= de=20
to be broken.
=20
=20
Like only use immutables as dictionary keys.
=20
Fair enough, but don;t go advising newbies to do this.

=20
How about something like this.
=20
Because of the extra precautions one has to take when
using mutables as hash keys, we advise newbies
to stick with immutable keys until they have gathered
enough knowledge and experience to adequatly weight
the pro and cons of a mutable key solution against
an immutable key solution.

knowledgeable and experienced users know when to ignore the rules.

Then why seems there to be so few acknowledgement that these rules
may indeed be broken by users. My experience is that anyone who suggests
so runs the risk of being branded a (python) heretic.

 

[Just]

Than you haven't looked hard enough.

All of these can get unexpected behaviour because of the
assignment-doesn't-copy semantics. The same semantics
that can cause problems if you work with mutable dictionary
keys.[/QUOTE]

Again, the difference is:

1. assigning mutable objects *can* cause unexpected behavior
(however, it's a useful feature, everyone using Python
for longer than a day or two knows this, and then it's
*expected* behavior.

2. mutating dict keys *does* *always* cause problems.
(unless you use an identity hash/cmp)

It's nonsense to forbid 1) since it's a useful feature. It's useful to
forbid ("discourage") 2) since mutating dict keys is seldom useful (and
when it is, Python lets you support it in your own objects).

Just

 

[Steve Holden]

I you want to play it like this

http://redwing.hutman.net/~mreed/warriorshtm/crybaby.htm

Hey, I'm not crying, or trying to play it any particular way. It's a
free Internet, and you are (more than) welcome to your opinions.
Personally I'd say I'm actually more of a combination of
http://redwing.hutman.net/~mreed/warriorshtm/diplomat.htm and
http://redwing.hutman.net/~mreed/warriorshtm/evilclown.htm, with added
touches of http://redwing.hutman.net/~mreed/warriorshtm/garble.htm and
possibly http://redwing.hutman.net/~mreed/warriorshtm/toxicgranny.htm

Whereas I also detect touches of
http://redwing.hutman.net/~mreed/warriorshtm/ferouscranus.htm in your
posts. It must be nice to *never* be wrong . Your resemblance to
http://redwing.hutman.net/~mreed/warriorshtm/filibuster.htm, however,
makes me wish you could be right rather more succinctly.

Mostly, though, I was trying to say that I found your nitpicking
insistence on terminological exactitude, even when giving advice to
those new to the language, both inappropriate and tedious in the extreme.

Have a nice day

regards
Steve

 

[Craig Ringer]

knowledgeable and experienced users know when to ignore the rules.

+1 QOTW

One of the nice things is that Python permits you to do exactly that
where appropriate while avoiding forcing you to do gruesome things to
get a job done.

I think the classic example of your statement is the use of 'goto' in C
and C++ code. Don't use goto - except when there's no other sensible way
to make your code clear. For example, goto and Py_XECREF seem to be very
handy for cleanup after detecting an exception when working with the
Python/C API.

That said, I do think "the rules" deserve consideration and respect -
they're usually there because of many others' experience over time. It's
interesting to learn those lessons first hand, but it's nice to be able
to avoid repeating every single one of them.

 

[Antoon Pardon]

All of these can get unexpected behaviour because of the
assignment-doesn't-copy semantics. The same semantics
that can cause problems if you work with mutable dictionary
keys.

Again, the difference is:

1. assigning mutable objects *can* cause unexpected behavior
(however, it's a useful feature, everyone using Python
for longer than a day or two knows this, and then it's
*expected* behavior.

2. mutating dict keys *does* *always* cause problems.
(unless you use an identity hash/cmp)[/QUOTE]

3 mutating an item in a sorted list *does* *always* cause problems

4 mutating an item in a heap queue *does* *always* cause problems

It's nonsense to forbid 1) since it's a useful feature. It's useful to
forbid ("discourage") 2) since mutating dict keys is seldom useful (and
when it is, Python lets you support it in your own objects).

Yes mutating dict keys is seldom usefull. But again you are conflating
mutable dict keys with mutating dict keys. No mutables as dict keys
means you can't mutate the object even if it is not a key. But whether
assigning mutable objects is usefull or not doesn't depend on whether
the object will also be usefull as a key or not. You treat this as
if an obejct is always a key or never and then decide mutable objects
as dict keys is not usefull because mutating keys is not usefull.

More specific the Decimal class is mutable and usable as dict key.

 

[Antoon Pardon]

Antoon Pardon wrote:

Mostly, though, I was trying to say that I found your nitpicking
insistence on terminological exactitude, even when giving advice to
those new to the language, both inappropriate and tedious in the extreme.

I think it is appropiate because not all new to the language are new to
programming and even newbees have a right to know how it really is. Otherwise
after some time you get experienced users who don't know the fact. In
this case for example there are a number of people who flat out assert that
muatble dict keys in pyhthon is impossible.

 

[Steven Bethard]

In this case for example there are a number of people who flat out
assert that muatble dict keys in pyhthon is impossible.

If you run into any of these folks, please point them to:

http://www.python.org/moin/DictionaryKeys

It's a pretty good summary of one of the more recent threads on this
topic by Peter Maas and myself. If you feel like your point is not
covered[1], feel free to add some additional material. (That is, after
all, the point of a wiki.)

Steve

[1] I assume that you feel this way because you have repeated
essentially the same point about 5 times in this thread. The wiki is a
chance to make your statement once, and then simply post a pointer once
per appropriate thread. It might save you some time...

 

[Simon Brunning]

3 mutating an item in a sorted list *does* *always* cause problems

No, it doesn't. It might cause the list no longer to be sorted, but
that might or might no be a problem.

More specific the Decimal class is mutable and usable as dict key.

Decimal objects are immutable, so far as I know.
False

 

[Antoon Pardon]

No, it doesn't. It might cause the list no longer to be sorted, but
that might or might no be a problem.

Than in the same vain I can say that mutating a key in a dictionary
doesn't always cause problems either. Sure it may probably make a
key unaccessible directly, but that might or might not be a problem.

Decimal objects are immutable, so far as I know.

True

 

[Nick Coghlan]

More specific the Decimal class is mutable and usable as dict key.

It's *meant* to be immutable though. The fact that we used __slots__ instead of
__setattr__ to implement the immutability, so you *can* overwrite the slot
variables if you really want to is merely an artifact of the current Python
implementation.

The limited mutability bug will disappear in Python 2.5, so it's not a good
example of a 'mutable' dict key (especially given that the only way to mutate it
is to modify private variables directly).

And, as I've stated previously, if the issue of sane mutable keys in
dictionaries and sets really bugs you so much - implement identity_dict and
identity_set in C and lobby for their inclusion in the collections module.

On the more general point of "don't use mutable objects with non-identity based
comparisons as dictionary keys", try teaching students for a while (or listen to
those who have):

When stating useful general principles, it is never, ever worth it to get into
the quibbly little details about exceptions to the principle. If students ask,
admit that they exist, but point out that the exceptions are rare, and not worth
worrying about at that point in their learning.

Only when one is aware of the reasons for a principle, can one be aware of good
reasons not to follow it

Cheers,
Nick.

 

[Antoon Pardon]

It's *meant* to be immutable though. The fact that we used __slots__ instead of
__setattr__ to implement the immutability, so you *can* overwrite the slot
variables if you really want to is merely an artifact of the current Python
implementation.

The limited mutability bug will disappear in Python 2.5, so it's not a good
example of a 'mutable' dict key (especially given that the only way to mutate it
is to modify private variables directly).

And, as I've stated previously, if the issue of sane mutable keys in
dictionaries and sets really bugs you so much - implement identity_dict and
identity_set in C and lobby for their inclusion in the collections module.

I'm not bugged by its absence in Python. I'm bugged by the attitude
about them. But anyway I'm thinking about implementing a safe dictionary
that will copy whenever it would otherwise allow the risk of mutating a
key.

On the more general point of "don't use mutable objects with non-identity based
comparisons as dictionary keys", try teaching students for a while (or listen to
those who have):

What kind of students?

I have implemented a hash table when I was a student and its
implementation allowed the use of 'mutable' objects as a key
without a problem. It simply always made copies when appropiate
and didn't allow external access to the keys. So although the
key objects were 'mutable' there was no way a user could accidently
mutate a key.

So don't use a mutable as a dictionary key isn't so much a dictionary
limitation in general but a specific limitation of the python implementation.

And yes I understand, the current implenatation is the result of the
fact that the same dictionaries are used internally for variables in
scopes and attributes in objects and the fact that no copies are
involved gives a boost to performance. But it could be argued that
providing these same dictionaries with those semantics to the users
was a premature optimisation.

When stating useful general principles, it is never, ever worth it to get into
the quibbly little details about exceptions to the principle. If students ask,
admit that they exist, but point out that the exceptions are rare, and not worth
worrying about at that point in their learning.

But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.

I don't like it when a good rule of thumb because of implementation
limitations is sold as a general principle.

 

[Steve Holden]

Op 2005-01-18, Nick Coghlan schreef <[email protected]>: [...]
But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.

Sorry, but it *is* a general principle, adduced from the potential
pitfalls available to inexperienced programmers when breaking the principle.

I don't like it when a good rule of thumb because of implementation
limitations is sold as a general principle.

So, since you are so good at nit-picking, perhaps you will explain the
difference between "rule of thumb" and "general principle".

preferably-in-less-than-three-thousand-words-ly y'rs - steve

 

[John Lenton]

Then why seems there to be so few acknowledgement that these rules
may indeed be broken by users. My experience is that anyone who suggests
so runs the risk of being branded a (python) heretic.

First you learn the basics, then you think you're knowledgeable and
experienced, then you learn the rules, then you become one with the
rules, and then you may break them.

Most people suggesting these things haven't gotten past step #3. Using
Craig's parallel to C's goto, every and all newbie using gotos should
be lambasted: even if the use might be correct for the problem they
are trying to solve, the reasons for its correctness are far too
complex for them to grasp. But really, in practically any system, the
rules are generalizations, and they exist because the particulars are
too delicate to trust the unexperienced. The small print is
unprintable.

He dicho.

--
John Lenton ([email protected]) -- Random fortune:
Yo siempre seré el futuro Nobel. Debe ser una tradición escandinava.
-- Jorge Luis Borges. (1899-1986) Escritor argentino.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)

iD8DBQFB7RuVgPqu395ykGsRAi6fAJ42W5rGkkqc6NaQUjxDPbsjUGZragCgs27C
sr5B5HIe+LXvYaxjqyc2PQg=
=L6xP
-----END PGP SIGNATURE-----

 

[Antoon Pardon]

Op 2005-01-18, Nick Coghlan schreef <[email protected]>: [...]
But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.

Sorry, but it *is* a general principle, adduced from the potential
pitfalls available to inexperienced programmers when breaking the principle.

But if these inexperienced programmers would have used dictionaries
that were implemented more safely, those pitfalls would have been
avoided. So are the pitfalls caused by breaking the principle
or by using an unsafe dictionary implementation?

So, since you are so good at nit-picking, perhaps you will explain the
difference between "rule of thumb" and "general principle".

A rule of thumb is context sensitive. If circumstances change,
so do the rules of thumb. Principles have a broader field
of application.

IMO there is nothing principally wrong with using a mutable object
as a dictionary key. But avoiding doing so is a good rule of
thumb if you have a python-like implementation of a dictionary.

 

[David Bolen]

Than in the same vain I can say that mutating a key in a dictionary
doesn't always cause problems either. Sure it may probably make a
key unaccessible directly, but that might or might not be a problem.

Well, I'd definitely consider an inaccessible key as constituting a
problem, but I don't think that's a good analogy to the list case.

With the dictionary, the change can (though I do agree it does not
have to) interfere with proper operation of the dictionary, while a
list that is no longer sorted still functions perfectly well as a
list. That is, I feel "problems" are more guaranteed with a
dictionary since we have affected base object behavior, whereas sorted
is not an inherent attribute of the base list type but something the
application is imposing at a higher level.

For example, I may choose to have an object type that is mutable (and
not worthy for use as a dictionary key) but maintains a logical
ordering so is sortable. I see no problem with sorting a list of such
objects, and then walking that list to perform some mutation to each
of the objects, even if along the way the mutation I am doing results
in the items so touched no longer being in sorted order. The act of
sorting was to provide me with a particular sequence of objects, but
aside from that fact, the list continues to perform perfectly well as
a list even after the mutations - just no longer delivering objects in
sorted order.

-- David

 

[Bengt Richter]

I'm not bugged by its absence in Python. I'm bugged by the attitude
about them. But anyway I'm thinking about implementing a safe dictionary
that will copy whenever it would otherwise allow the risk of mutating a
key.

I encourage you to do it. It should be a rich learning experience.
My bet is you will discover something about the real requirements of
what you are "thinking about implementing" ;-)

IANAP, but ISTM that after a certain point, trying to get made-my-point satisfaction
in a thread like this is more like OCD symptomatology than n.g. dialog ;-)

What kind of students?

I have implemented a hash table when I was a student and its
implementation allowed the use of 'mutable' objects as a key
without a problem. It simply always made copies when appropiate
and didn't allow external access to the keys. So although the
key objects were 'mutable' there was no way a user could accidently
mutate a key.

This is word play IMO. What you describe is effectively using the mutables
to construct immutable keys for actual use, not using immutable keys. Having
the immutable (because of access restriction) constructed keys allows
you to check on their consistency with their source any time you want,
but that doesn't change the fact that you have created an alternative dictionary
implementation with immutable keys made immutable by copying and access restriction.
If you update your dictionary when a key no longer matches its source data, you
are just deleting an immutable-by-special-means key and entering the associated
value in association with a new immutable-by-special-means key.

I can produce OCD symptoms too ;-)

So don't use a mutable as a dictionary key isn't so much a dictionary
limitation in general but a specific limitation of the python implementation.

And yes I understand, the current implenatation is the result of the
fact that the same dictionaries are used internally for variables in
scopes and attributes in objects and the fact that no copies are
involved gives a boost to performance. But it could be argued that
providing these same dictionaries with those semantics to the users
was a premature optimisation.

If you see a sign like

+--------------------------------->
| WILD GOOSE CHASE THIS WAY ->
+----------------------------->
||
|| \|/
|| =o=
|| /|\
..`.||,..__|_,.

Do you feel that you must make sure?
The sign painter may only be indicating what his own experience was, after all.
But if it was a core developer's experience, it may be prudent to bet on another trail
-- unless you have an extremely interesting hunch. Then you should follow your passion
and have your own experience, and you may wind up being able to contribute something
(even if only an exclamation point on the sign ;-)

But don't use mutable keys is not a general principle. It is a principle
introduced by the limitations of the python implementations.

That is so in your mind, and it is so given certain interpretations of the words
you are using, but to others "mutable keys" may be a contradiction in terms, because
they do not use the words in the same sense as you. If you can't see alternative conceptual
constructs you are stuck, and if we can't agree on the meaning of words for a given dialog,
we won't be communicating very well. The difficulty is getting people to recognize their
implicit assumptions and definitions ;-)

I don't like it when a good rule of thumb because of implementation
limitations is sold as a general principle.

People take short cuts in expressing themselves, just as you do.
E.g., you say "mutable key" when you really mean a value that will
remain constant while you are using it as a dictionary key. The "safe" dictionary
that you are "thinking about" will apparently get its "safety" by ensuring that
the "mutable key" can't be "mutated" -- or do you want to discuss which copy
is the real "key" ;-) (I don't ;-)

Regards,
Bengt Richter