Thoughts on PEP284

[Stephen Horne]

Some more looping thoughts - this time on integer for loops...

There may be some hint towards PEP284 (integer for loops) in a review
of ideas from other languages, but I'm damned if i can figure it out.
I spent some time thinking about it and couldn't find anything that
would cover the issue.

All I came up with was the recognition that some other languages have
'for' and 'foreach' loops. But Python went down the route of using
'for' for looping over lists/iterators a long time ago. Maybe a new
keyword or combination could help, such as...

for range i in 0 <= i < 10 :

or...

for range 0 <= i < 10 :

or dropping the PEP284 concept in favor of something C based...

for i yield i = 0; i < 10; i += 1 :

but I don't much like any of those, to be honest.


I did come up with something, though...

I doubt the need for exclusive ranges in integer for loops. I also
doubt the need for switching between different range systems
(inclusive, exclusive, half-open). IMO there is more confusion than
anything down those routes.

Although I haven't had problems with inclusive loops (Basic, Ada) my
preference is toward half-open loops if a choice needs to be made. And
it occurs to me that Python programmers do specify integer ranges in a
half-open way on a regular basis - when doing slicing on lists and
strings.

Perhaps the 'int' and 'long' types could support a slicing to generate
a list or iterator over the range?

If so, we could write...

for i in int[0:10] :
...

If the syntax creates an iterator rather than a list, it could even be
used as follows...

for i in long[0:] :
...
break if <condition> :
...

We could have some basic in-step looping with...

for i, j in zip(int[0:], int[len(list)-1::-1] :
...
break if i > l :
...

but this is probably an argument against.

What would be good, however, is that it can be useful outside of
looping for free - list comprehensions, for instance...

[i*i for i in int[0:10]]

It would need some slight restrictions compared with normal slicing
(it needs to know the value to start at, even if not where to stop)
but what the hell. At least I'm not going to be accused of trying to
turn Python into another language - AFAIK this slice-of-a-type idea is
original. This could be my first time being accused of trying to turn
Python into a language that doesn't exist yet, of course ;-)

 

[Sean Ross]

Some more looping thoughts - this time on integer for loops...

Hi.
How about the following:

for i in 0...10:
# suite

where 0...10 works something like xrange(10). So you get i=0 then 1 then 2
.... then 9. To get 0 thru 10, inclusive, you'd need to say 0...11 (Ruby has
0..10, but I doubt that'd pass muster here). This suggestion,
unfortunately, does not provide step control and I can't think of a clean
way to introduce it other than via keyword:

for i in 0...10 by 2:
# suite

which will also be shot down, I suspect, as would

for i in 0 to 10 by 2:

for i in 0 thru 10 by 2:

etc.

Perhaps generators could grow a by() method so that you could control how
you step thru the iteration, something like this:

for i in 0...10.by(2):
# suite

Whatever.

Leaving the ellipsis aside, you could add methods to int/long (also ala
Ruby),

for i in 0.upto(10):
# suite

# to count down
for i in 10.downto(0):
# suite

where the methods upto() and downto() are generators with an optional step
argument, i.e.

def upto(self, stop, step=1):
# suite


And, after all of that, you could say, well we already have

for i in range(10):
# suite

If only range was a generator function...but that'd break code, and you can
use xrange(), and some day it will(may) be, and ..., and ..., yada, yada,
yada.

Whatever.

 

[Stephen Horne]

Hi.
How about the following:

for i in 0...10:
# suite

where 0...10 works something like xrange(10). So you get i=0 then 1 then 2
... then 9. To get 0 thru 10, inclusive, you'd need to say 0...11 (Ruby has
0..10, but I doubt that'd pass muster here).

Personally, I think the conventional '..' has more chance than your
'...'. Basically, the idea of '..' specifying an inclusive range is a
familiar idea to most people, even those who haven't used languages
like Ruby or Ada that use that notation. Having a syntax that looks
similar but means something just slightly different is likely to be
seriously error prone and confusing.

I'm certainly not against '..' in the conventional sense (I like
half-open ranges, but I'm not a fundamentalist and inclusive ranges
have worked perfectly well in Basic and Ada (and IIRC Pascal, Modula 2
and others).

Actually, the main reason I didn't mention it myself is that I figure
there must be some reason for its being rejected in the past. I can't
seriously believe that it has never been proposed.

Leaving the ellipsis aside, you could add methods to int/long (also ala
Ruby),

for i in 0.upto(10):
# suite

# to count down
for i in 10.downto(0):
# suite

I'm not sure about this - I'm bouncing between "that's neat!" and
"yuck!", but the "yuck!" part may just mean I need to get used to it.

 

[David Eppstein]

Some more looping thoughts - this time on integer for loops...

Hi.
How about the following:

for i in 0...10:
# suite

where 0...10 works something like xrange(10).[/QUOTE]

I can't find the message you're replying to, so am responding here only
to yours.

I personally am uninterested in an integer for-loop syntax that is
limited to the unintuitiveness of range and xrange's arguments --
they're great for looping forwards through the indices into a list, not
so great for almost anything else. If that's all the loop syntax can
do, why not just keep the current "for i in range(10):" syntax?
"Explicit is better than implicit."

But anyway, PEP 284 has been kind of moribund since Guido dissed it in
his Parade of PEPs. I'm not sure what (if anything) it would take to
revive it, but convoluted syntax like "for i in 0...10.by(2):" is
probably not it. "for i in 0:10:2:" would be more Pythonic, but has a
little ambiguity problem ("for i in 0:10:2" without the colon seems like
it should execute the statement "2" ten times) and again doesn't add
much in readability or power to what we can already do with range.

 

[Paul Foley]

Some more looping thoughts - this time on integer for loops...

There may be some hint towards PEP284 (integer for loops) in a review
of ideas from other languages, but I'm damned if i can figure it out.
I spent some time thinking about it and couldn't find anything that
would cover the issue.

All I came up with was the recognition that some other languages have
'for' and 'foreach' loops. But Python went down the route of using
'for' for looping over lists/iterators a long time ago. Maybe a new
keyword or combination could help, such as...

for range i in 0 <= i < 10 :

for range 0 <= i < 10 :

or dropping the PEP284 concept in favor of something C based...

for i yield i = 0; i < 10; i += 1 :

but I don't much like any of those, to be honest.

for i from 0: # open-ended range
...

for i below 10: # == range(10)
...

for i from 5 below 10: # == range(5,10)
...

for i to 10: # == range(11)
...

for i to 10 by 2: # == range(0,10,2)
...

for i from 5 to 10: # == range(5,11)
...

for i from 10 downto 2: # == range(10,1,-1)
...

for i = x then f(i): # arbitrary update expression
...

etc., etc.


[http://www.lispworks.com/reference/HyperSpec/Body/m_loop.htm: for-as-clause]

--
Let me control a planet's oxygen supply and I don't care who makes the
laws.

(setq reply-to
(concatenate 'string "Paul Foley " "<mycroft" '(#\@) "actrix.gen.nz>"))

 

[Sean Ross]

Hi.
How about the following:

for i in 0...10:
# suite

where 0...10 works something like xrange(10).

I can't find the message you're replying to, so am responding here only
to yours.

I personally am uninterested in an integer for-loop syntax that is
limited to the unintuitiveness of range and xrange's arguments --
they're great for looping forwards through the indices into a list, not
so great for almost anything else. If that's all the loop syntax can
do, why not just keep the current "for i in range(10):" syntax?[/QUOTE]

Hi.
Right.

Quote from me:
"""
And, after all of that, you could say, well we already have

for i in range(10):
# suite
....
and you can use xrange()
....

yada, yada, yada.

Whatever.
"""

"Explicit is better than implicit."

Um. Actually, for i in 0...10 _is_ explicit. It explicitly asks for the
integers from 0 thru 9, just as xrange(10) does, but with different syntax.

"where 0...10 works something like xrange(10). "
me, again

I'm familiar with seeing ellipsis used to denote ranges (from textbooks,
other programming languages, and general every day writings), so, to me it's
looks explicit. (Although, I would usually read it as the numbers from 0 to
10, inclusive). But, again, whatever.

But anyway, PEP 284 has been kind of moribund since Guido dissed it in
his Parade of PEPs. I'm not sure what (if anything) it would take to
revive it,

Sure. I've no intentions of pursuing this syntax proposal. The PEP was
brought up. A syntax suggestion was made. I didn't much care for those, so I
made others. My interest ends there.

but convoluted syntax like "for i in 0...10.by(2):" is
probably not it.

Right.

"This suggestion ... does not provide step control and I can't think of a
clean way to introduce it ..."
me

the by() suggestion was a demonstration of how I could not "think of a clean
way to introduce" step control for the ellipsis based syntax.

"for i in 0:10:2:" would be more Pythonic, but has a
little ambiguity problem ("for i in 0:10:2" without the colon seems like
it should execute the statement "2" ten times) and again doesn't add
much in readability or power to what we can already do with range.

Right.

 

[Stephen Horne]

"for i in 0:10:2:" would be more Pythonic, but has a
little ambiguity problem ("for i in 0:10:2" without the colon seems like
it should execute the statement "2" ten times) and again doesn't add
much in readability or power to what we can already do with range.

That's pretty close to my suggestion, which - in brief - was...

for i in int [0:10:2] :
...

that is, allow the 'int' type object to be sliced, and return an
iterator as the result - not just for the for loop, but generally
(though optimising the 'for' loop usage to avoid physically creating
the iterator might be a good idea).

This shouldn't be too wierd as Pythonistas are already familiar with
slicing - it's just that a type is being sliced rather than a list or
string. But to anyone familiar with computer science, a set of values
is a key feature of any abstract data type anyway. It just so happens
that the set of integers is conveniently sliceable (as long as the
start value is specified).

True, it is mostly just an alternative notation for range or xrange.
But the integer for loops thing is something that never seems to go
away. PEP284 itself quotes four previous PEPs on broadly the same
issue.

The rejected PEP204 is uncomfortably close to my (and your)
suggestions, but while the difference is small I think it is
significant. It is reasonable to think of a type as a potentially
sliceable set of values, and practical to implement providing the type
has certain properties such as being discrete.

Also, one extra feature is that the loop can be infinite (which range
and xrange cannot achieve)...

for i in int [0:] :
...
if condition : break
...

 

[M-a-S]

I guess this should be a matter of optimization. Why don't the Python
compiler recognize 'for <var> in [x]range(<from>,<till>):'? It should be
pretty easy. Microsoft does marvels optimizing loops in C. Why an open
source project like Python can't do it?

M-a-S

 

[David Eppstein]

There may be some hint towards PEP284 (integer for loops) in a review
of ideas from other languages, but I'm damned if i can figure it out.
I spent some time thinking about it and couldn't find anything that
would cover the issue.

All I came up with was the recognition that some other languages have
'for' and 'foreach' loops. But Python went down the route of using
'for' for looping over lists/iterators a long time ago. Maybe a new
keyword or combination could help, such as...

for range i in 0 <= i < 10 :

[/QUOTE]

The idea behind the specific syntax of PEP284 was simply the following
observation: one way of reading "for i in iterator" is that it loops
over the values for which the expression "i in iterator" is true.
What types of expressions other than the "in" operator could be
substituted in the same context and do something useful?

 

[Stephen Horne]

I guess this should be a matter of optimization. Why don't the Python
compiler recognize 'for <var> in [x]range(<from>,<till>):'? It should be
pretty easy. Microsoft does marvels optimizing loops in C. Why an open
source project like Python can't do it?

For all I know it does. Complaints about the 'xrange' notation are
normally about the syntax, not the efficiency. For instance, take this
quote from Guidos parade of PEPs...

"""
PEP 284 - Integer for-loops - Eppstein, Ewing
Yet another way to address the fact that some people find

for i in range(10):

too ugly.
"""

The word is "ugly", not "slow".

 

[Sean Ross]

Personally, I think the conventional '..' has more chance than your
'...'.

Hi.
Actually, I think '...' is more conventional notation for expressing a range
of numbers. Like when you want to show a sequence:

# odd numbers greater than 3, but less than 257
5, 7, 9, 11, ..., 255, 257

That sort of thing.

Also, '...' is one dot further away from a single dot '.', than is '..'. Is
that important? Maybe. I figure it takes away one argument:

"a..b looks too much like a.b"

by saying

"Well, a...b doesn't"

Of course, a#%$^^#b doesn't either....

Anyway ...

I'm not sure about this - I'm bouncing between "that's neat!" and
"yuck!", but the "yuck!" part may just mean I need to get used to it.

I'm more:
I've seen it in Ruby and Smalltalk (where integers have such methods), and
it's not '...' , nor is it range(), so that might be an alternative. Whether
it is a good alternative ... <shrug>. It doesn't add new syntax, only new
methods to existing objects. But, it does not appear to provide any marked
improvement over range (or xrange). So, it's neither here nor there for me.

In my own language, were I to make one, I would use:

for i in start to stop by step:
# suite

But, that's not going to happen in this language; neither do I expect it to,
nor would I ask for it (I'd make my own, if I really wanted it).
Still, it's interesting to play with different formulations, to see if they
do or do not work well.

 

[Stephen Horne]

The idea behind the specific syntax of PEP284 was simply the following
observation: one way of reading "for i in iterator" is that it loops
over the values for which the expression "i in iterator" is true.
What types of expressions other than the "in" operator could be
substituted in the same context and do something useful?[/QUOTE]

OK - that makes sense, but I don't think I'd naturally read '0 <= i <
10' as an iterator without a good deal of prompting. Even with the
explanation, I still feel the need for a different (or extra) keyword
to emphasise the difference.

 

[Raymond Hettinger]

I guess this should be a matter of optimization. Why don't the Python

compiler recognize 'for <var> in [x]range(<from>,<till>):'? It should be
pretty easy. Microsoft does marvels optimizing loops in C. Why an open
source project like Python can't do it?

Yes, it's true that C's for(i=0;i<n;i++){action} runs faster than Python's
for i in xrange(n). However, once you've converted "i" to a PyInt
object and done signal checking / thread switching on each iteration, the
gap closes quite a bit. Python's for loops are very lightweight and their
overhead tends to be tiny in comparison to any real work being done
inside the loop.

To get true marvels, you need to do something like psyco and be able
dynamically analyze the loop to determine that "i" need not be a PyInt
but can be stored, incremented, and accessed as a C long.

Short of dynamic analysis and native code generation, your best bet
is to look at stack frame creation and argument passing when calling
a function written in pure python. The next best bet is to look at
optimizing method lookup and calls. In comparision, the performance
of the for-loop is a red herring.


Raymond Hettinger


P.S. If you don't need to use "i" in the body of the loop, the high-speed
pure python version becomes: for _ in itertools.repeat(None, n): body

If the loop is boundless, then "while 1" is even better (it has practically
*no* overhead).

 

[Stephen Horne]

Hi.
Actually, I think '...' is more conventional notation for expressing a range
of numbers. Like when you want to show a sequence:

# odd numbers greater than 3, but less than 257
5, 7, 9, 11, ..., 255, 257

That sort of thing.

Good point, but I think the commas are essential to making that clear.
Without the commas, it simply doesn't look like the same thing.

I've seen it in Ruby and Smalltalk (where integers have such methods), and
it's not '...' , nor is it range(), so that might be an alternative. Whether
it is a good alternative ... <shrug>. It doesn't add new syntax, only new
methods to existing objects.

That is also an advantage to my slicing idea - it only requires a new
method (__getitem__ IIRC) be implemented for one object (the builtin
type object called 'int').

 

[Dennis Lee Bieber]

Paul Foley fed this fish to the penguins on Monday 22 September 2003
09:18 pm:

for i from 0: # open-ended range
...

for i below 10: # == range(10)
...

for i from 5 below 10: # == range(5,10)
...

for i to 10: # == range(11)
...

for i to 10 by 2: # == range(0,10,2)
...

for i from 5 to 10: # == range(5,11)
...

for i from 10 downto 2: # == range(10,1,-1)
...

for i = x then f(i): # arbitrary update expression
...

etc., etc.

Maybe you'd like the REXX statement?

DO i = <start> TO <end> BY <step> WHILE <cond>

Though where either the control variable side or the conditional
variable side are optional (and you can use UNTIL instead of WHILE), if
you don't need a control variable inside the loop you can use flat
count "DO 100", or "DO FOREVER"

I think the AREXX implementation actually allowed /both/ WHILE and
UNTIL on the same statement...

--

 

[Daniel Dittmar]

for i in int [0:10:2] :
...

that is, allow the 'int' type object to be sliced, and return an
iterator as the result - not just for the for loop, but generally
(though optimising the 'for' loop usage to avoid physically creating
the iterator might be a good idea).

Adding operators to types is always problematic because it defeats Pythons's
runtime checking. Assuming that integer slicing would be added to Python,
methods that would expect a list of integers would suddenly also work with
integers. In most cases, they would not work correctly, but you wouldn't get
a meaningful exception.

Also, one extra feature is that the loop can be infinite (which range
and xrange cannot achieve)...

for i in int [0:] :
...
if condition : break
...

I'm sure that you can think of a generator function that does exactly the
same.

Daniel

 

[Alex Martelli]

for i in int [0:10:2] :
...

that is, allow the 'int' type object to be sliced, and return an
iterator as the result - not just for the for loop, but generally
(though optimising the 'for' loop usage to avoid physically creating
the iterator might be a good idea).

Adding operators to types is always problematic because it defeats
Pythons's runtime checking. Assuming that integer slicing would be added
to Python, methods that would expect a list of integers would suddenly
also work with integers. In most cases, they would not work correctly, but
you wouldn't get a meaningful exception.

Careful: in Stephen's proposal, slicing would NOT work on *INTEGERS* --
rather, it would work on the *INT TYPE ITSELF*, which is a very, very
different issue. "Methods that would expect a list of integers", ONLY
do slicing on that list, AND get mistakenly passed the type object
'int' itself -- would work just perfectly, as if they had been passed
"a list of all non-negative integers" or xrange(sys.maxint+1).

Also, one extra feature is that the loop can be infinite (which range
and xrange cannot achieve)...

for i in int [0:] :

No way, Jose -- now THAT would break things (in admittedly rare
cases -- a method expecting a list and NOT providing an upper bound
in the slicing). I'd vote for this to be like int[0:sys.maxint+1]
(i.e., last item returned is sys.maxint).

I'm sure that you can think of a generator function that does exactly the
same.

Right, that's pretty trivial, and itertools.count() already does it anyway.


What Stephen's proposal lacks is rigorous specs of what happens for all
possible slices -- e.g int[0:-3] isn't immediately intuitive, IMHO;-).


Alex

 

[Gerrit Holl]

Also, one extra feature is that the loop can be infinite (which range
and xrange cannot achieve)...

for i in int [0:] :

No way, Jose -- now THAT would break things (in admittedly rare
cases -- a method expecting a list and NOT providing an upper bound
in the slicing). I'd vote for this to be like int[0:sys.maxint+1]
(i.e., last item returned is sys.maxint).

This one breaks the int/long unification. In an ideal unification
of ints and longs, sys.maxint would not exist any more, and the
upper bound would be seemingly random.

Gerrit.

 

[Alex Martelli]

Also, one extra feature is that the loop can be infinite (which range
and xrange cannot achieve)...

for i in int [0:] :

No way, Jose -- now THAT would break things (in admittedly rare
cases -- a method expecting a list and NOT providing an upper bound
in the slicing). I'd vote for this to be like int[0:sys.maxint+1]
(i.e., last item returned is sys.maxint).

This one breaks the int/long unification. In an ideal unification
of ints and longs, sys.maxint would not exist any more, and the
upper bound would be seemingly random.

If this is the "party line" -- that not providing an upper bound
when indexing int should generate an infinitely looping iterator --
then I think the risk of nasty bugs would probably be too high, and
I would reluctantly -1 the otherwise nice proposal. I'd rather
have the missing upper bound be an error in this case (and rely on
itertools.count for very explicit building of infinitely looping
iterators) than "easily create infinities" in response to typos;-).


Alex

 

[Sean Ross]

Here's a quick hack of an int class that supports iteration using the slice
notation, plus simple iteration on the class itself (for i in int: ...).
This can certainly be improved, and other issues need to be addressed, but I
just wanted to see what Stephen's idea would look like in practice. (Not
bad. Actually, pretty good.)

# using python 2.2.2
from __future__ import generators

class xint(int):
class __metaclass__(type):
def __iter__(cls):
return cls.__getitem__()
def __getitem__(cls, index=None):
if hasattr(index, "start"):
for i in range(index.start, index.stop, index.step):
yield i
elif isinstance(index, int):
yield index
elif index is None:
i = 0
while True:
yield i
i += 1
else:
raise Exception
__getitem__ = classmethod(__getitem__)

print "iteration on int"
for i in xint:
if i >= 10:
break
print i

print "\niteration on int slice"
for i in xint[0:22:2]:
print i



#
# OUTPUT
#
iteration on int
0
1
2
3
4
5
6
7
8
9

iteration on int slice
0
2
4
6
8
10
12
14
16
18
20