Pre-PEP: Dictionary accumulator methods

[Paul Rubin]

What about no name at all for the scalar case:

a['hello'] += 1
a['bye'] -= 2

I like this despite the minor surprise that it works even when
a['hello'] is uninitialized.

 

[John Machin]

I would like to get everyone's thoughts on two new dictionary

methods:

+1 for each.

PROBLEMS BEING SOLVED
---------------------

The readability issues with the existing constructs are:

* They are awkward to teach, create, read, and review.
* Their wording tends to hide the real meaning (accumulation).
* The meaning of setdefault() 's method name is not self-evident.

+1 to EACH of the above 3 points.

A question directed to the folk who had to look up "tally" in the
dictionary: Which dictionary includes "setdefault", "updateBy", etc?

The performance issues with the existing constructs are:
[MANY]

the performance improvement matches the readability
improvement.
Agreed.



ISSUES

+3 for tally !!!

appendtolist is better than appendlist

The appendlist() method is not as versatile as setdefault() which can be used
with other object types (perhaps for creating dictionaries of dictionaries).
However, most uses I've seen are with lists.

My use cases for tally:
(1) Yes the text-book "word" frequency gig applied in production
data-matching etc applications
(2) quick stats like from SQL "group by" e.g.
customer.tally(state)
customer_value.tally(state, dollar_value) # real or *DECIMAL*

Use cases for appendlist: many; in general, how else do you implement a
one-to-many relationship in memory??

 

[Bengt Richter]

I would like to get everyone's thoughts on two new dictionary methods:

def count(self, value, qty=1):
try:
self[key] += qty
except KeyError:
self[key] = qty

def appendlist(self, key, *values):
try:
self[key].extend(values)
except KeyError:
self[key] = list(values)

How about an efficient duck-typing value-incrementer to replace both? E.g. functionally like:

class xdict(dict):

... def valadd(self, key, incr=1):
... try: self[key] = self[key] + type(self[key])(incr)
... except KeyError: self[key] = incr

A big problem with this is that there are reasonable use cases for both
d.count(key, <some integer>)
and
d.appendlist(key, <some integer>)

Word counting is an obvious use for the first. Consolidating a list of key, value pairs where the
values are ints requires the second.

Combining count() and appendlist() into one function eliminates the second possibility.

I don't see a "big problem" ;-)

d.addval doesn't eliminate the functionalities if you want them. You just have to spell them
d.addval(key, <some integer>)
and
d.addval(key, [<some integer>])
respectively.

My example interactive stuff in a previous post shows both of these using xd['x'] and xd.['y']:
""" {}

Default integer 1 arg creates initial int value {'x': 1}

Explicit list arg create initial list value {'y': [0, 1, 2], 'x': 1}

Explicit int increment adds to existing int

>>> xd.valadd('x', 100)
>>> xd['x']

101

Explicit list arg extends existing list with contents of increment list
which you can of course optionally use with a length-1 list to achieve the .append effect

>>> xd.valadd('y', range(3,6))
>>> xd['y']

[0, 1, 2, 3, 4, 5]
"""

Granted, d.appendlist will result in more efficient code, since the temp list arg
[<some integer>] does not have to be created and the type of the existing dict value
does not have to be tested as generally. OTOH, you can create and extend tuple
or even custom object values using valadd, and extend with alternate sequences that
get converted to the existing type if its contructor knows how to accept alternatives.

So I think you are not prevented from doing anything. IMO Raymond's Zen concerns
are the ones to think about first, and then efficiency, which was one of the motivators
in the first place ;-)

Regards,
Bengt Richter

 

[Reinhold Birkenfeld]

methods:

+1 for each.


+1 to EACH of the above 3 points.

A question directed to the folk who had to look up "tally" in the
dictionary: Which dictionary includes "setdefault", "updateBy", etc?

Are you kidding? If you know what "set" and "default" means, you will be
able to guess what "setdefault" means. Same goes for updateBy.

Of course, I had to look up setdefault in the Python docs the first time
I ran across it, but in the case of tally, I would have to look up both
in the Python docs and in my dictionary.

Reinhold

 

[Raymond Hettinger]

[Bengt Richter]

IMO Raymond's Zen concerns
are the ones to think about first, and then efficiency, which was one of the motivators
in the first place ;-)

Well said.

I find the disassembly (presented in the first post) to be telling. The
compiler has done a great job and there is no fluff -- all of those steps have
been specified by the programmer and he/she must at some level be aware of every
one of them (pre-instantiation, multiple method lookups and calls, multiple
dictionary accesses, etc). That is too bad because the intent could have been
stated atomically: d.appendlist(k, v). Instead, the current idiom turns you
away from what you want done and focuses the attention on how it is done:
d.setdefault(k, []).append(v). That is too many steps for what should be an
atomic operation (in the eyes of the programmer and code readers).

Likewise, d.tally(k) is as atomic as this expression can get. Any other steps,
added verbiage, new types, extra arguments, or whatnot are an unnecessary waste
of brain cells.


Raymond Hettinger

 

[Paul Rubin]

I find the disassembly (presented in the first post) to be telling.
The compiler has done a great job and there is no fluff -- all of
those steps have been specified by the programmer and he/she must at
some level be aware of every one of them (pre-instantiation,
multiple method lookups and calls, multiple dictionary accesses, etc).

If the compiler can do some type inference, it can optimize out those
multiple calls pretty straightforwardly.

 

[John Machin]

John Machin wrote:
Are you kidding? If you know what "set" and "default" means, you will be
able to guess what "setdefault" means. Same goes for updateBy.

No I'm not kidding -- people from some cultures have no difficulty at
all in mentally splitting up "words" like "setdefault" or the German
equivalent of "Danubesteamnavigationcompany'sdirector'swife"; others
from other cultures where agglutinisation is not quite so rife will
have extreme difficulty.

And "Updateby" sounds like a village somewhere in the Danelaw

 

[David Eppstein]

The rationale is to replace the awkward and slow existing idioms for
dictionary
based accumulation:

d[key] = d.get(key, 0) + qty
d.setdefault(key, []).extend(values)

In simplest form, those two statements would now be coded more readably as:

d.count(key)
d.appendlist(key, value)

In their multi-value forms, they would now be coded as:

d.count(key, qty)
d.appendlist(key, *values)

The error messages returned by the new methods are the same as those returned
by
the existing idioms.

The get() method would continue to exist because it is useful for
applications
other than accumulation.

The setdefault() method would continue to exist but would likely not make it
into Py3.0.

The other dictionary based accumulation I've used but don't see here is
with sets as dictionary values, i.e.
dictionary.setdefault(key,set()).add(value).

I suppose it would be possible to appendlist then make a set from the
list, but if you were to take that minimalist philosophy to an extreme
you wouldn't need count either, you could just appendlist then use len.

 

[David Eppstein]

Also, in all of my code base, I've not run across a single opportunity to use
something like unionset().

In my code, this would be roughly tied with appendlist for frequency of
use.

 

[Ron]

I would like to get everyone's thoughts on two new dictionary methods:

def count(self, value, qty=1):
try:
self[key] += qty
except KeyError:
self[key] = qty

def appendlist(self, key, *values):
try:
self[key].extend(values)
except KeyError:
self[key] = list(values)

-1 form me.

I'm not very glad with both of them ( not a naming issue ) because i
think that the dict type should offer only methods that apply to each
dict whatever it contains. count() specializes to dict values that are
addable and appendlist to those that are extendable. Why not
subtractable, dividable or right-shiftable? Because of majority
approval? I'm mot a speed fetishist and destroying the clarity of a
very fundamental data structure for speedup rather arbitrary
accumulations seems to be a bad idea. I would move this stuff in a
subclass.

Regards Kay

-1 for me too.

I agree with this. The other dictionary functions are all data
nuetral. Adding special methods to handle data should be in a
durrived class and not a built in. imho.

# Roll your own specialized dictionaries.

class NumDict( dict):
def count(self, key, qty=1):
try:
self[key] += qty
except KeyError:
self[key] = qty
def appendlist(self, key, *values):
try:
self[key].extend(values)
except KeyError:
self[key] = list(values)

mydict = NumDict()

n = 0
for k in list('abcdefg'):
n += 1
mydict[k] = n

print mydict
# {'a': 1, 'c': 3, 'b': 2, 'e': 5, 'd': 4, 'g': 7, 'f': 6}

mydict.count('c')
mydict['e'] = ['a']
mydict.appendlist('e', 'bcdef')
print mydict
# {'a': 1, 'c': 4, 'b': 2, 'e': ['a', 'bcdef'], 'd': 4, 'g': 7, 'f':
6}


This isn't that hard to do. I don't think the extra methods should be
added to the base class.

Ron

 

[George Sakkis]

-1 form me.

I'm not very glad with both of them ( not a naming issue ) because i
think that the dict type should offer only methods that apply to each
dict whatever it contains. count() specializes to dict values that are
addable and appendlist to those that are extendable. Why not
subtractable, dividable or right-shiftable? Because of majority
approval? I'm mot a speed fetishist and destroying the clarity of a
very fundamental data structure for speedup rather arbitrary
accumulations seems to be a bad idea. I would move this stuff in a
subclass.

Regards Kay

+1 on this. The new suggested operations are meaningful for a subset of all valid dicts, so they
should not be part of the base dict API. If any version of this is approved, it will clearly be an
application of the "practicality beats purity" zen rule, and the justification for applying it in
this case instead of subclassing should better be pretty strong; so far I'm not convinced though.

George

 

[Michael Spencer]

I would like to get everyone's thoughts on two new dictionary methods:

def count(self, value, qty=1):
try:
self[key] += qty
except KeyError:
self[key] = qty

def appendlist(self, key, *values):
try:
self[key].extend(values)
except KeyError:
self[key] = list(values)

These undoubtedly address common cases, which are unsatisfactory when spelled
using setdefault. However...

Use of these methods implicitly specializes the dictionary. The methods are
more-or-less mutually exclusive i.e., it would be at least strange to use count
and appendlist on the same dictionary. Moreover, on many dictionary instances,
the methods would fail or produce meaningless results.

This seems to be at odds with the other methods of built-in container types
which can be meaningfully applied, no matter what the types of the contents.
(There may be exceptions, but I can't think of any at the moment)

Does anyone else think this is a problem?

Michael

 

[George Sakkis]

I would like to get everyone's thoughts on two new dictionary methods:

def count(self, value, qty=1):
try:
self[key] += qty
except KeyError:
self[key] = qty

def appendlist(self, key, *values):
try:
self[key].extend(values)
except KeyError:
self[key] = list(values)

These undoubtedly address common cases, which are unsatisfactory when spelled
using setdefault. However...

Use of these methods implicitly specializes the dictionary. The methods are
more-or-less mutually exclusive i.e., it would be at least strange to use count
and appendlist on the same dictionary. Moreover, on many dictionary instances,
the methods would fail or produce meaningless results.

This seems to be at odds with the other methods of built-in container types
which can be meaningfully applied, no matter what the types of the contents.
(There may be exceptions, but I can't think of any at the moment)

Does anyone else think this is a problem?

Michael

Yep, at least three more people in this thread:
- http://tinyurl.com/4bsdf
- http://tinyurl.com/3seqx
- http://tinyurl.com/6db27

George

 

[Alexander Schmolck]

The rationale is to replace the awkward and slow existing idioms for dictionary
based accumulation:

d[key] = d.get(key, 0) + qty
d.setdefault(key, []).extend(values)

In simplest form, those two statements would now be coded more readably as:

d.count(key)
d.appendlist(key, value)

Yuck.

The relatively recent "improvement" of the dict constructor signature
(``dict(foo=bar,...)``) obviously makes it impossible to just extend the
constructor to ``dict(default=...)`` (or anything else for that matter) which
would seem much less ad hoc. But why not use a classmethod (e.g.
``d=dict.withdefault(0)``) then?

Or, for the first and most common case, just a bag type?


'as

 

[John Machin]

+1 on this. The new suggested operations are meaningful for a subset of all valid dicts, so they
should not be part of the base dict API. If any version of this is

approved, it will clearly be an

application of the "practicality beats purity" zen rule, and the

justification for applying it in

this case instead of subclassing should better be pretty strong; so

far I'm not convinced though.

My background: I've been subclassing dict since this was possible, to
provide not only a count/tally method but also a DIY intern method.
Before that I just copied dictmodule.c (every release!) and diffed and
hacked about till I had a mydict module.

*any* version? Could we make you happy by having a subclass
TotallySinfulDict provided as part of the core? You don't have to use
it -- come to think of it, you don't have to use a sinful method in the
base dict. You could even avert your gaze when reading the
documentation.

The justification for having it in the core: it's in C, not in Python,
it gets implemented and maintained (a) ONCE (b) by folk like the timbot
and Raymond instead of having (a) numerous versions lashed up (b) by
you and me and a whole bunch of n00bz and b1ffz

 

[Beni Cherniavsky]

The rationale is to replace the awkward and slow existing idioms for dictionary
based accumulation:

d[key] = d.get(key, 0) + qty
d.setdefault(key, []).extend(values)

Indeed not too readable. The try..except version is better but is too
verbose. There is a simple concept underneath of assuming a default value and
we need "one obvious" way to write it.

Yuck.

-1 from me too on these two methods because they only add "duct tape" for the
problem instead of solving it. We need to improve upon `dict.setdefault()`,
not specialize it.

The relatively recent "improvement" of the dict constructor signature
(``dict(foo=bar,...)``) obviously makes it impossible to just extend the
constructor to ``dict(default=...)`` (or anything else for that matter) which
would seem much less ad hoc. But why not use a classmethod (e.g.
``d=dict.withdefault(0)``) then?

You mean giving a dictionary a default value at creation time, right?

Such a dictionary could be used very easily, as in <gasp>Perl::

foreach $word ( @words ) {
$d{$word}++; # default of 0 assumed, simple code!
}

</gasp>. You would like to write::

d = dict.withdefault(0) # or something
for word in words:
d[word] += 1 # again, simple code!

I agree that it's a good idea but I'm not sure the default should be specified
at creation time. The problem with that is that if you pass such a dictionary
into an unsuspecting function, it will not behave like a normal dictionary.
Also, this will go awry if the default is a mutable object, like ``[]`` - you
must create a new one at every access (or introduce a rule that the object is
copied every time, which I dislike). And there are cases where in different
points in the code operating on the same dictionary you need different default
values.

So perhaps specifying the default at every point of use by creating a proxy is
cleaner::

d = {}
for word in words:
d.withdefault(0)[word] += 1

Of course, you can always create the proxy once and still pass it into an
unsuspecting function when that is actually what you mean.

How should a dictionary with a default value behave (wheter inherently or a
proxy)?

- ``d.__getattr__(key)`` never raises KeyError for missing keys - instead it
returns the default value and stores the value as `d.setdefult()` does.
This is needed for make code like::

d.withdefault([])[key].append(foo)

to work - there is no call of `d.__setattr__()`, so `d.__getattr__()` must
have stored it.

- `d.__setattr__()` and `d.__delattr__()` behave normally.

- Should ``key in d`` return True for all keys? It is desiarable to have
*some* way to know whether a key is really present. But if it returns False
for missing keys, code that checks ``key in d`` will behave differently from
normally equivallent code that uses try..except. If we use the proxy
interface, we can always check on the original dictionary object, which
removes the problem.

- ``d.has_key(key)`` must do whatever we decide ``key in d`` does.

- What should ``d.get(key, [default])`` and ``d.setdefault(key, default)``
do? There is a conflict between the default of `d` and the explicitly given
default. I think consistency is better and these should pretend that `key`
is always present. But either way, there is a subtle problem here.

- Of course `iter(d)`, `d.items()` and the like should only see the keys
that are really present (the alternative inventing an infinite amount of
items out of the blue is clearly bogus).

If the idea that the default should be specified in every operation (creating
a proxy) is accepted, there is a simpler and more fool-proof solution: the
ptoxy will not support anything except `__getitem__()` and `__setitem__()` at
all. Use the original dictionary for everything else. This prevents subtle
ambiguities.

Or, for the first and most common case, just a bag type?

Too specialized IMHO. You want a dictionary with any default anyway. If you
have that, what will be the benefit of a bag type?

 

[George Sakkis]

approved, it will clearly be an
justification for applying it in
far I'm not convinced though.

My background: I've been subclassing dict since this was possible, to
provide not only a count/tally method but also a DIY intern method.
Before that I just copied dictmodule.c (every release!) and diffed and
hacked about till I had a mydict module.

*any* version? Could we make you happy by having a subclass
TotallySinfulDict provided as part of the core? You don't have to use
it -- come to think of it, you don't have to use a sinful method in the
base dict. You could even avert your gaze when reading the
documentation.

The justification for having it in the core: it's in C, not in Python,
it gets implemented and maintained (a) ONCE (b) by folk like the timbot
and Raymond instead of having (a) numerous versions lashed up (b) by
you and me and a whole bunch of n00bz and b1ffz

I believe it was pretty clear that I'm not against a new dict extension, in the core or in the
standard library; the proposed functionality is certainly useful and it would be most welcome. I
just don't find it appropriate for the existing base dict because it is not applicable to *every*
dictionary. As for the "you don't have to use feature X if you don't like it" argument, it's rarely
relevant in language design.

George

 

[Kay Schluehr]

+1 on this. The new suggested operations are meaningful for a subset of all
valid dicts, so they
should not be part of the base dict API. If any version of this is

approved, > it will clearly be an

application of the "practicality beats purity" zen rule, and the
justification for applying it in
this case instead of subclassing should better be pretty strong; so far I'm
not convinced though.

George

It is bad OO design, George. I want to be a bit more become more
specific on this and provide an example:

Let be <intdict> a custom subclass of dict that stores only ints as
keys as well as values:

class intdict(dict):
def __setitem__(self, key, value):
assert isinstance(key, int) and isinstance(value, int)
dict.__setitem__(self,key,value)

or in Python3000 typeguard fashion:

class intdict(dict):
def __setitem__(self, key:int, value:int):
dict.__setitem__(self,key,value)

But <intdict> still overloads appendlist() i.e. a method that does not
work for any intdict is still part of it's public interface! This is
really bad design and should not be justified by a "practicality beats
purity" wisdom which should be cited with much care but not
carelesness.

Maybe also the subclassing idea I introduced falls for short for the
same reasons. Adding an accumulator to a dict should be implemented as
a *decorator* pattern in the GoF meaning of the word i.e. adding an
interface to some object at runtime that provides special facilities.

Usage:
False

This could be generalized to other fundamental data structures as well.

Regards Kay

 

[Michael Spencer]

Maybe also the subclassing idea I introduced falls for short for the
same reasons. Adding an accumulator to a dict should be implemented as
a *decorator* pattern in the GoF meaning of the word i.e. adding an
interface to some object at runtime that provides special facilities.

Usage:



False

This could be generalized to other fundamental data structures as well.

Regards Kay

Or similarly, something like the 'reversed' view of a sequence:

I could imagine a class: accumulator(mapping, default, incremetor) such that:

>>> my_tally = accumulator({}, 0, operator.add) or
>>> my_dict_of_lists = accumulator({}, [], list.append) or
>>> my_dict_of_sets = accumulator({}, set(), set.add)

then: .accumulate(key, value) "does the right thing" in each case.

a bit cumbersome, because of having to specify the accumulation method, but
avoids adding methods to, or sub-classing dict

Michael

 

[Michele Simionato]

I am surprised nobody suggested we put those two methods into a
separate module (say dictutils or even UserDict) as functions:

from dictutils import tally, listappend

tally(mydict, key)
listappend(mydict, key, value)

I am -1 about a specific subclass of dict in the standard library, I
would not mind about a few new functions
in an utility module.

Michele Simionato