max(), sum(), next()

[Steven D'Aprano]

On Sat, 06 Sep 2008 11:22:07 -0700, Mensanator wrote:

[...]

Ok. But the problem is they DID in SQL: x + Null = Null.

Sheesh. That's not a problem, because Python is not trying to be a
dialect of SQL.

If you want a NULL object, then there are recipes on the web that will
give you one. Then all you need to do is call sum(alist or [NULL]) and it
will give you the behaviour you want.


[...]

Here's a real world example (no ivory tower stuff):

An oil refinery client has just excavated a big pile of dirt to lay a
new pipeline. [snip details]
Can't I just use a sum of 0 to tell me when data is missing? No, because
in some cases the reporting limit of undetected compounds is set to 0.

You can't use a sum of 0 to indicate when data is missing, full stop. The
data may require 15 tests when only 3 have actually been done:

sum([1.2e-7, 9.34e-6, 2.06e-8])

Missing data and a non-zero sum. How should sum() deal with that?

The answer is that sum() can't deal with that. You can't expect sum() to
read your mind, know that there should be 15 items instead of 3, and
raise an error. So why do you expect sum() to read your mind and
magically know that zero items is an error, especially when for many
applications it is NOT an error?

The behaviour you want for this specific application is unwanted,
unnecessary and even undesirable for many other applications. The
solution is for *you* to write application-specific code to do what your
application needs, instead of relying on a general purpose function
magically knowing what you want.

 

[Mensanator]

On Sat, 06 Sep 2008 11:22:07 -0700, Mensanator wrote:

[...]

Ok. But the problem is they DID in SQL: x + Null = Null.

Sheesh. That's not a problem, because Python is not trying to be a
dialect of SQL.

And yet, they added a Sqlite3 module.

If you want a NULL object, then there are recipes on the web that will
give you one. Then all you need to do is call sum(alist or [NULL]) and it
will give you the behaviour you want.

Actualy, I already get the behaviour I want. sum([1,None])
throws an exception. I don't see why sum([]) doesn't throw
an exception also (I understand that behaviour is by design,
I'm merely pointing out that the design doesn't cover every
situation).

[...]

Here's a real world example (no ivory tower stuff):

An oil refinery client has just excavated a big pile of dirt to lay a
new pipeline. [snip details]
Can't I just use a sum of 0 to tell me when data is missing? No, because
in some cases the reporting limit of undetected compounds is set to 0.

You can't use a sum of 0 to indicate when data is missing, full stop.

Exactly. That's why I would prefer sum([]) to raise an
exception instead of giving a false positive.

The
data may require 15 tests when only 3 have actually been done:

sum([1.2e-7, 9.34e-6, 2.06e-8])

Biggest problem here is that it is often unknown just
how many records you're supposed to get from the query,
so we can't tell that a count of 3 is supposed to be 15.

Missing data and a non-zero sum. How should sum() deal with that?

That's a seperate issue and I'm not saying it should as
long as the list contains actual numbers to sum.
sum([1.2e-7, 9.34e-6, 2.06e-8, None]) will raise an
exception, as it should. But what types are contained
in []?

The answer is that sum() can't deal with that. You can't expect sum() to
read your mind, know that there should be 15 items instead of 3, and
raise an error. So why do you expect sum() to read your mind and
magically know that zero items is an error, especially when for many
applications it is NOT an error?

For the simple reason it doesn't have to read your mind,
a mechanism has already been built into the function: start
value. For those situations where an empty list is desired
to sum to 0, you could use sum(alist,0) and use sum(alist) for
those cases where summing an empty list is meaningless.
Shouldn't you have to explicitly tell sum() how deal with
situations like empty lists rather than have it implicitly
assume a starting value of 0 when you didn't ask for it?

The behaviour you want for this specific application is unwanted,
unnecessary and even undesirable for many other applications. The
solution is for *you* to write application-specific code to do what your
application needs, instead of relying on a general purpose function
magically knowing what you want.

Does division magically know what you want? No, it raises an
exception when you do something like divide by 0. Isn't it
Pythonic to not write a litany of tests to cover every
possible case, but instead use try:except?

But try:except only works if the errors are recognized.
And sum() says that summing an empty list is NEVER an error
under ANY circumstance. That may be true in MOST cases, but
it certainly isn't true in ALL cases.

 

[Patrick Maupin]

On Sep 6, 11:05 pm, Steven D'Aprano <st...@REMOVE-THIS-

And yet, they added a Sqlite3 module.

Does that mean that, because there is an 'os' module, Python is trying
to compete with Linux and Windows?

This is starting to feel like a troll, but JUST IN CASE you are really
serious about wanting to get work done with Python, rather than
complaining about how it is not perfect, I offer the following snippet
which will show you how you can test the results of a sum() to see if
there were any items in the list:
.... pass
....

zero = MyZero()
x=sum([], zero)
isinstance(x,MyZero) True
x = sum([1,2,3], zero)
isinstance(x,MyZero) False

 

[Gabriel Genellina]

Actualy, I already get the behaviour I want. sum([1,None])
throws an exception. I don't see why sum([]) doesn't throw
an exception also (I understand that behaviour is by design,
I'm merely pointing out that the design doesn't cover every
situation). [...]
Exactly. That's why I would prefer sum([]) to raise an
exception instead of giving a false positive.

The built in behavior can't be good for every usage. Nobody prevents you from defining yoru own function tailored to your own specs, like this:

def strict_sum(items):
items = iter(items)
try:
first = items.next()
except StopIteration:
raise ValueError, "strict_sum with empty argument"
return sum(items, first)

Tweak as needed. Based on other posts I believe your Python skills are enough to write it on your own, so I don't see why you're complaining so hard about the current behavior.

 

[Luis Zarrabeitia]

Actualy, I already get the behaviour I want. sum([1,None])
throws an exception. I don't see why sum([]) doesn't throw
an exception also

If you take a "start value" and add to it every element of a list, should the
process fail if the list is empty? If you don't add anything to the start value,
you should get back the start value.

Python's sum is defined as sum(sequence, start=0). If sum were to throw an
exception with sum([]), it should also throw it with sum([], start=0), wich
makes no sense.

 

[Mensanator]

En Sun, 07 Sep 2008 14:30:09 -0300, Mensanator <[email protected]> escribi�:

Actualy, I already get the behaviour I want. sum([1,None])
throws an exception. I don't see why sum([]) doesn't throw
an exception also (I understand that behaviour is by design,
I'm merely pointing out that the design doesn't cover every
situation). [...]
Exactly. That's why I would prefer sum([]) to raise an
exception instead of giving a false positive.

The built in behavior can't be good for every usage. Nobody prevents you from defining yoru own function tailored to your own specs, like this:

def strict_sum(items):
� � items = iter(items)
� � try:
� � � � first = items.next()
� � except StopIteration:
� � � � raise ValueError, "strict_sum with empty argument"
� � return sum(items, first)

Tweak as needed. Based on other posts I believe your Python skills are enough to write it on your own, so I don't see why you're complaining so hard about the current behavior.

I'm not complaining about the behaviour anymore, I just don't like
being told I'm wrong when I'm not.

But I think I've made my point, so there's no point in harping on
this anymore.

 

[Mensanator]

Does that mean that, because there is an 'os' module, Python is trying
to compete with Linux and Windows?

I wasn't thinking "compete", rather "complement". Python obviously
wants to be a player in the SQL market, so you would think it
would be in Python's interest to know how SQL behaves, just as it's in
Python's interest for the os module to know how BOTH Linnux and
Windows work.

This is starting to feel like a troll,

It wasn't intended to be.

but JUST IN CASE you are really
serious about wanting to get work done with Python, rather than
complaining about how it is not perfect,

Things never change if no one ever speaks up.

I offer the following snippet
which will show you how you can test the results of a sum() to see if
there were any items in the list:

Thanks. I'll drop this from this point on.

... � � pass
...

zero = MyZero()
x=sum([], zero)
isinstance(x,MyZero) True
x = sum([1,2,3], zero)
isinstance(x,MyZero)

False- Hide quoted text -

- Show quoted text -

 

[Mensanator]

� � � � Which is an interface TO an embedded/stand-alone SQL-based RDBM
engine; it does not turn Python into a dialect of SQL -- Python does not
process the SQL, it gets passed to the engine for SQL data processing.

But that's only half the story. The other half is data returned
as a result of SQL queries. And that's something Python DOES process.
And sometimes that processed data has to be inserted back into the
database. We certainly don't want Python to process the data in a way
that the database doesn't expect.

When I see a potential flaw (such as summing an empty list to 0),
should I just keep quiet about it, or let everyone know?

Well, now they know, so I'll shut up about this from now on, ok?

 

[Boris Borcic]

(ii) If A is a subset of B then we should have
max(A) <= max(B). This requires that max(empty set)
be something that's smaller than everything else.
So we give up on that.

Er, what about instances of variations/elaborations on

class Smaller(object) : __cmp__ = lambda *_ : -1

?

Cheers, BB

 

[castironpi]

Er, what about instances of variations/elaborations on

class Smaller(object) : __cmp__ = lambda *_ : -1

?

Cheers, BB

You still don't have the property max(X) is in X.

And it's the equivalent of a special builtin constant for max on the
empty set.

 

[Boris Borcic]

You still don't have the property max(X) is in X.

Frankly, I would favor order-independence over that property.

compare max(X) for

1) X = [set([1]),set([2])]

and

2) X = [set([2]),set([1])]

Shouldn't then max and min in fact return lub and glb, despite their names ? In
the case X is a non-empty finite set/list of totally ordered values,
max(X)==lub(X) and min(X)=glb(X) in any case.

And it's the equivalent of a special builtin constant for max on the
empty set.

Of course (except the object might have other uses, who knows). So what ?

Cheers, BB

 

[Mensanator]

Actualy, I already get the behaviour I want. sum([1,None])
throws an exception. I don't see why sum([]) doesn't throw
an exception also

If you take a "start value" and add to it every element of a list, should the
process fail if the list is empty?
No.

If you don't add anything to the start value,
you should get back the start value.
Agree.


Python's sum is defined as sum(sequence, start=0).

That's the issue.

If sum were to throw an
exception with sum([]), it should also throw it with sum([], start=0), wich
makes no sense.

Given that definition, yes. But is the definition correct
in ALL cases? Are there situations where the sum of an empty
list should NOT be 0? Of course there are.

Can sum() handle those cases? No, it can't, I have to write
my own definition if I want that behaviour. There's no reason
why sum([]) and sum([],0) have to mean the same thing at the
exclusion of a perfectly valid alternative definition.

But that's the way it is, so I have to live with it.

But that's not conceeding that I'm wrong.

 

[Terry Reedy]

Are there situations where the sum of an empty
list should NOT be 0? Of course there are.

Python Philosopy (my version, for this discussion):
Make normal things easy; make unusual or difficult things possible.

Application:
Sum([]) == 0 is normal (90+% of cases). Make that easy (as it is).
For anything else:
if seq: s = sum(s, base)

Can sum() handle those cases?

The developers choose what they thought would be most useful across the
spectrum of programmers and programs after some non-zero amount of
debate and discussion.

> No, it can't, I have to write
my own definition if I want that behaviour.

Or wrap your calls. In any case, before sum was added as a convenience
for summing numbers, *everyone* has to write their own or use reduce.

Sum(s) replaces reduce(lambda x,y: x+y, s, 0), which was thought to be
the most common use of reduce. Sum(s,start) replaces the much less
common reduce(lambda x,y: x+y, s, start).

Reduce(S, s), where S = sum function, raises an exception on empty s.
So use that and you are no worse off than before.

However, a problem with reduce(S,s) is that it is *almost* the same as
reduce(S,s,0). So people are sometimes tempted to omit 0, especially if
they are not sure if the call might be reduce(S,0,s) (as one argument
says it should be -- but that is another post). But if they do, the
program fails, even if it should not, if and when s happens to be empty.

There's no reason
why sum([]) and sum([],0) have to mean the same thing at the
exclusion of a perfectly valid alternative definition.

'Have to', no reason. 'Should', yes there are at least three reasons.
1. Python functions generally return an answer rather than raise an
exception where there is a perfectly valid answer to return.
2. As a general principle, something that is almost always true should
not need to be stated over and over again. This is why, for instance,
we have default args.
3. As I remember, part of the reason for adding sum was to eliminate the
need (with reduce) to explicitly say 'start my sum at 0' in order to
avoid buggy code. In other words, I believe part of the reason for
sum's existence is to avoid the very bug-inviting behavior you want.

Terry Jan Reedy

 

[Mensanator]

Are there situations where the sum of an empty
list should NOT be 0? Of course there are.

Python Philosopy (my version, for this discussion):
   Make normal things easy; make unusual or difficult things possible..

Application:
   Sum([]) == 0 is normal (90+% of cases).  Make that easy (as it is).
   For anything else:
     if seq: s = sum(s, base)

Can sum() handle those cases?

The developers choose what they thought would be most useful across the
spectrum of programmers and programs after some non-zero amount of
debate and discussion.

 >  No, it can't, I have to write

my own definition if I want that behaviour.

Or wrap your calls.  In any case, before sum was added as a convenience
for summing numbers, *everyone* has to write their own or use reduce.

Sum(s) replaces reduce(lambda x,y: x+y, s, 0), which was thought to be
the most common use of reduce.  Sum(s,start) replaces the much less
common reduce(lambda x,y: x+y, s, start).

Reduce(S, s), where S = sum function, raises an exception on empty s.
So use that and you are no worse off than before.

What am I doing wrong?

S = sum
S

s = [1,2,3]
type(s)

reduce(S,s)

Traceback (most recent call last):
File "<pyshell#13>", line 1, in <module>
reduce(S,s)
TypeError: 'int' object is not iterable
Traceback (most recent call last):
File "<pyshell#14>", line 1, in <module>
reduce(S,s,0)
TypeError: 'int' object is not iterable

6

s=[]
reduce(lambda x,y:x+y,s)

Traceback (most recent call last):
File "<pyshell#17>", line 1, in <module>
reduce(lambda x,y:x+y,s)
TypeError: reduce() of empty sequence with no initial value

This is supposed to happen. But doesn't reduce(S,s) work
when s isn't empty?

However, a problem with reduce(S,s) is that it is *almost* the same as
reduce(S,s,0).  So people are sometimes tempted to omit 0, especially if
they are not sure if the call might be reduce(S,0,s) (as one argument
says it should be -- but that is another post).  But if they do, the
program fails, even if it should not, if and when s happens to be empty.

There's no reason
why sum([]) and sum([],0) have to mean the same thing at the
exclusion of a perfectly valid alternative definition.

'Have to', no reason.  'Should', yes there are at least three reasons.
1. Python functions generally return an answer rather than raise an
exception where there is a perfectly valid answer to return.
2. As a general principle, something that is almost always true should
not need to be stated over and over again.  This is why, for instance,
we have default args.
3. As I remember, part of the reason for adding sum was to eliminate the
need (with reduce) to explicitly say 'start my sum at 0' in order to
avoid buggy code.  In other words, I believe part of the reason for
sum's existence is to avoid the very bug-inviting behavior you want.

Terry Jan Reedy

 

[Terry Reedy]

What am I doing wrong?

[snip]

Taking me too literally out of context. I meant the sum_of_2 function
already given in the example above, as you eventually tried.

def S(x,y): return x+y

Sorry for the confusion.

....

reduce(lambda x,y:x+y,s)

6

s=[]
reduce(lambda x,y:x+y,s)

Traceback (most recent call last):
File "<pyshell#17>", line 1, in <module>
reduce(lambda x,y:x+y,s)
TypeError: reduce() of empty sequence with no initial value

These two are exactly what I meant.

This is supposed to happen. But doesn't reduce(S,s) work
when s isn't empty?

It did. You got 6 above. The built-in 'sum' takes an iterable, not a
pair of numbers.

tjr

 

[Boris Borcic]

Hi,

Quoting Laszlo Nagy <[email protected]>:
...
Even better:

help(sum) shows

===
sum(...)
sum(sequence, start=0) -> value
Returns the sum of a sequence of numbers (NOT strings) plus
the value
of parameter 'start'. When the sequence is empty, returns start.
===

so the fact that sum([]) returns zero is just because the start value
is zero...
sum([],object()) would return an object().

BTW, the original code:

sum(s for s in ["a", "b"] if len(s) > 2)

wouldn't work anyway... it seems that sum doesn't like to sum strings:

sum(['a','b'],'')

<type 'exceptions.TypeError'>: sum() can't sum strings [use
''.join(seq) instead]

Yes which is a bit bad anyway. I don't think hard wiring it is such a
nice idea. You know, walks like a duck, smells like a duck...
If it makes sense to handle things differently for performance, then
please have it doing it silently, e.g. when it detects strings just
use join() internally.

Cheers
Tino

+1

''.join is horrible. And it adds insult to injury that S.join(S.split(T)) != T
as a rule. The interpreter has no business to patronize us into this shamefully
contorted neighborhood while it understands what we want.

Cheers, BB

 

[Boris Borcic]

Tino Wildenhain wrote: [...]

sum(['a','b'],'')

<type 'exceptions.TypeError'>: sum() can't sum strings [use
''.join(seq) instead]

Yes which is a bit bad anyway. I don't think hard wiring it is such a
nice idea. You know, walks like a duck, smells like a duck...
If it makes sense to handle things differently for performance, then
please have it doing it silently, e.g. when it detects strings just
use join() internally.

Cheers
Tino

+1

''.join is horrible. And it adds insult to injury that
S.join(S.split(T)) != T as a rule. The interpreter has no business to
patronize us into this shamefully contorted neighborhood while it
understands what we want.

What makes ''.join particularly horrible is that we find ourselves forced to use
it not only for concatenating arbitrary-length strings in a list, but also to
convert to a str what's already a sequence of single characters. IOW string
types fail to satisfy a natural expectation for any S of sequence type :

S == type(S)(item for item in S) == type(S)(list(S))

And this, even though strings are sequence types deep-down-ly enough that they
achieve to act as such in far-fetched corner cases like

(lambda *x : x)(*'abc')==('a','b','c')

....and even though strings offer not one but two distinct constructors that play
nicely in back-and-forth conversions with types to which they are much less
closely related, ie.

'1j' == repr(complex('1j') == str(complex('1j'))
1j == complex(repr(1j)) == complex(str(1j))

Not-so-cheerfully-yours, BB