Do you have real-world use cases for map's None fill-in feature?

[Raymond Hettinger]

I am evaluating a request for an alternate version of itertools.izip()
that has a None fill-in feature like the built-in map function:
[('a', '1'), ('b', '2'), ('c', '3'), (None, '4'), (None, '5')]

The movitation is to provide a means for looping over all data elements
when the input lengths are unequal. The question of the day is whether
that is both a common need and a good approach to real-world problems.
The answer to the question can likely be found in results from other
programming languages or from real-world Python code that has used
map's None fill-in feature.

I scanned the docs for Haskell, SML, and Perl and found that the norm
for map() and zip() is to truncate to the shortest input or raise an
exception for unequal input lengths. I scanned the standard library
and found no instances where map's fill-in feature was used. Likewise,
I found no examples in all of the code I've ever written.

The history of Python's current zip() function serves as another
indicator that the proposal is weak. PEP 201 contemplated and rejected
the idea as one that likely had unintended consequences. In the years
since zip() was introduced in Py2.0, SourceForge has shown no requests
for a fill-in version of zip().

My request for readers of comp.lang.python is to search your own code
to see if map's None fill-in feature was ever used in real-world code
(not toy examples). I'm curious about the context, how it was used,
and what alternatives were rejected (i.e. did the fill-in feature
improve the code).

Also, I'm curious as to whether someone has seen a zip fill-in feature
employed to good effect in some other programming language, perhaps
LISP or somesuch?

Maybe a few real-word code examples and experiences from other
languages will shed light on the question of whether lock-step
iteration has meaning beyond the length of the shortest matching
elements. If ordinal position were considered as a record key, then
the proposal equates to a database-style outer join operation (where
data elements with unmatched keys are included) and order is
significant. Does an outer-join have anything to do with lock-step
iteration? Is this a fundamental looping construct or just a
theoretical wish-list item? IOW, does Python really need
itertools.izip_longest() or would that just become a distracting piece
of cruft?


Raymond Hettinger


P.S. FWIW, the OP's use case involved printing files in multiple
columns:

for f, g in itertools.izip_longest(file1, file2, fillin_value=''):
print '%-20s\t|\t%-20s' % (f.rstrip(), g.rstrip())

The alternative was straight-forward but not as terse:

while 1:
f = file1.readline()
g = file2.readline()
if not f and not g:
break
print '%-20s\t|\t%-20s' % (f.rstrip(), g.rstrip())

 

[Paul Rubin]

I am evaluating a request for an alternate version of itertools.izip()
that has a None fill-in feature like the built-in map function:

I think finding different ways to write it was an entertaining
exercise but it's too limited in usefulness to become a standard
feature.

I do think some idiom ought to develop to allow checking whether an
iterator is empty, without consuming an item. Here's an idea:
introduce something like

iterator = check_empty(iterator)

where check_empty would work roughly like (untested):

def check_empty(iterator):
iclass = iterator.__class__
class buffered(iclass):
def __init__(self):
n = iter((self.next(),)) # might raise StopIteration
self.__save = chain(n, self)
def next(self):
return self.__save.next()
# all other operations are inherited from iclass

return buffered(iterator)

The idea is you get back a new iterator which yields the same stream
and supports the same operations as the old one, if the old one is
non-empty. Otherwise it raises StopIteration.

There are some obvious problems with the above:

1) the new iterator should support all of the old one's attributes,
not just inherit its operations
2) In the case where the old iterator is already buffered, the
constructor should just peek at the lookahead instead of making
a new object. That means that checking an iterator multiple times
won't burn more and more memory.

Maybe there is some way of doing the above with metaclasses but I've
never been able to wrap my head around those.

 

[Bengt Richter]

I am evaluating a request for an alternate version of itertools.izip()
that has a None fill-in feature like the built-in map function:
[('a', '1'), ('b', '2'), ('c', '3'), (None, '4'), (None, '5')]

I don't like not being able to supply my own sentinel. None is too common
[('a', '1'), ('b', '2'), ('c', '3'), (None(5), '4'), (None(5), '5')]

But seriously, standard sentinels for "missing data" and "end of data" might be nice to have,
and to have produced in appropriate standard contexts. Singleton string subclass
instances "NOD" and "EOD"? Doesn't fit with EOF=='' though.

The movitation is to provide a means for looping over all data elements
when the input lengths are unequal. The question of the day is whether
that is both a common need and a good approach to real-world problems.
The answer to the question can likely be found in results from other
programming languages or from real-world Python code that has used
map's None fill-in feature.

What about some semantics like my izip2 in
http://groups.google.com/group/comp.lang.python/msg/3e9eb63a1ddb1f46?hl=en

(which doesn't even need a separate name, since it would be backwards compatible)

Also, what about factoring sequence-related stuff into being methods or attributes
of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
which could be a substitute for izip and then some? Methods could be called via a returned
iterator before or after the first .next() call, to control various features, such as
sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
The point being to have a place to implement universal sequence stuff.

I scanned the docs for Haskell, SML, and Perl and found that the norm
for map() and zip() is to truncate to the shortest input or raise an
exception for unequal input lengths. I scanned the standard library
and found no instances where map's fill-in feature was used. Likewise,
I found no examples in all of the code I've ever written.

The history of Python's current zip() function serves as another
indicator that the proposal is weak. PEP 201 contemplated and rejected
the idea as one that likely had unintended consequences. In the years
since zip() was introduced in Py2.0, SourceForge has shown no requests
for a fill-in version of zip().

My request for readers of comp.lang.python is to search your own code
to see if map's None fill-in feature was ever used in real-world code
(not toy examples). I'm curious about the context, how it was used,
and what alternatives were rejected (i.e. did the fill-in feature
improve the code).

Also, I'm curious as to whether someone has seen a zip fill-in feature
employed to good effect in some other programming language, perhaps
LISP or somesuch?

ISTM in general there is a chicken-egg problem where workarounds are easy.
I.e., the real question is how many workaround situations there are
that would have been handled conveniently with a builtin feature,
and _then_ to see whether the convenience would be worth enough.

Maybe a few real-word code examples and experiences from other
languages will shed light on the question of whether lock-step
iteration has meaning beyond the length of the shortest matching
elements. If ordinal position were considered as a record key, then
the proposal equates to a database-style outer join operation (where
data elements with unmatched keys are included) and order is
significant. Does an outer-join have anything to do with lock-step
iteration? Is this a fundamental looping construct or just a
theoretical wish-list item? IOW, does Python really need
itertools.izip_longest() or would that just become a distracting piece
of cruft?

Even if there is little use for continuing in correct code, IWT getting
at the state of the iterator in an erroroneous situation would be a benefit.
Being able to see the result of the last attempt at gathering tuple elements
could help. (I can see reasons for wanting variations of trying all streams
vs shortcutting on the first to exhaust though).

Regards,
Bengt Richter

 

[Raymond Hettinger]

[Bengt Richter]

What about some semantics like my izip2 in
http://groups.google.com/group/comp.lang.python/msg/3e9eb63a1ddb1f46?hl=en

(which doesn't even need a separate name, since it would be backwards compatible)

Also, what about factoring sequence-related stuff into being methods or attributes
of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
which could be a substitute for izip and then some? Methods could be called via a returned
iterator before or after the first .next() call, to control various features, such as
sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
The point being to have a place to implement universal sequence stuff.

ISTM, these cures are worse than the disease ;-)

Even if there is little use for continuing in correct code, IWT getting
at the state of the iterator in an erroroneous situation would be a benefit.
Being able to see the result of the last attempt at gathering tuple elements
could help. (I can see reasons for wanting variations of trying all streams
vs shortcutting on the first to exhaust though).

On the one hand, that seems reasonable. On the other hand, I can't see
how to use it without snarling the surrounding code in which case it is
probably better to explicitly manage individual iterators within a
while loop.


Raymond

 

[Raymond Hettinger]

[Raymond Hettinger]
[Paul Rubin]

I think finding different ways to write it was an entertaining
exercise but it's too limited in usefulness to become a standard
feature.

Makes sense.

I do think some idiom ought to develop to allow checking whether an
iterator is empty, without consuming an item. Here's an idea:
introduce something like

iterator = check_empty(iterator)

There are so many varieties of iterator that it's probably not workable
to alter the iterator API for all of the them. In any case, a broad
API change like this would need its own PEP.

There are some obvious problems with the above:

1) the new iterator should support all of the old one's attributes,
not just inherit its operations
2) In the case where the old iterator is already buffered, the
constructor should just peek at the lookahead instead of making
a new object. That means that checking an iterator multiple times
won't burn more and more memory.

Maybe there is some way of doing the above with metaclasses but I've
never been able to wrap my head around those.

Metaclasses are unlikely to be of help because there are so many,
unrelated kinds of iterator -- most do not inherit from a common
parent.


Raymond

 

[Paul Rubin]

There are so many varieties of iterator that it's probably not workable
to alter the iterator API for all of the them. In any case, a broad
API change like this would need its own PEP.

The hope was that it wouldn't be an API change, but rather just a new
function dropped into the existing library, that could wrap any
existing iterator without having to change or break anything that's
already been written. Maybe the resulting iterator couldn't support
every operation, or maybe it could have a __getattr__ that delegates
everything except "next" to the wrapped iterator, or something. The
obvious implementation methods that I can see are very kludgy but
maybe something better is feasible. I defer to your knowledge about
this.

 

[Tim Peters]

[Raymond Hettinger]

...
I scanned the docs for Haskell, SML, and Perl and found that the norm
for map() and zip() is to truncate to the shortest input or raise an
exception for unequal input lengths.
...
Also, I'm curious as to whether someone has seen a zip fill-in feature
employed to good effect in some other programming language, perhaps
LISP or somesuch?

FYI, Common Lisp's `pairlis` function requires that its first two
arguments be lists of the same length. It's a strain to compare to
Python's zip() though, as the _intended_ use of `pairlis` is to add
new pairs to a Lisp association list. For that reason, `pairlis`
accepts an optional third argument; if present, this should be an
association list, and pairs from zipping the first two arguments are
prepended to it. Also for this reason, the _order_ in which pairs are
taken from the first two arguments isn't defined(!).

http://www.lispworks.com/documentation/HyperSpec/Body/f_pairli.htm#pairlis

For its intended special-purpose use, it wouldn't make sense to allow
arguments of different lengths.

 

[Bengt Richter]

[Bengt Richter]

What about some semantics like my izip2 in
http://groups.google.com/group/comp.lang.python/msg/3e9eb63a1ddb1f46?hl=en

(which doesn't even need a separate name, since it would be backwards compatible)

Also, what about factoring sequence-related stuff into being methods or attributes
of iter instances? And letting iter take multiple sequences or callable/sentinel pairs,
which could be a substitute for izip and then some? Methods could be called via a returned
iterator before or after the first .next() call, to control various features, such as
sentinel testing by 'is' instead of '==' for callable/sentinel pairs, or buffering n
steps of lookahead supported by a .peek(n) method defaulting to .peek(1), etc. etc.
The point being to have a place to implement universal sequence stuff.

ISTM, these cures are worse than the disease ;-)

Are you reacting to my turgidly rambling post, or to
...
('a', '1', 'A')
('b', '2', 'B')

Then after a backwards-compatible izip, if the iterator has
been bound, it can be used to continue, with sentinel sustitution:
...
('c', '<sentinel>', 'C')
('<sentinel>', '<sentinel>', 'D')

or optionally in sentinel substitution mode from the start:
...
('a', '1', 'A')
('b', '2', 'B')
('c', '<sentinel>', 'C')
('<sentinel>', '<sentinel>', 'D')

Usage-wise, this seems not too diseased to me, so I guess I want to make sure
this is what you were reacting to ;-)

(Implementation was just to hack together a working demo. I'm sure it can be improved upon ;-)

On the one hand, that seems reasonable. On the other hand, I can't see
how to use it without snarling the surrounding code in which case it is
probably better to explicitly manage individual iterators within a
while loop.

The above would seem to allow separation of concerns. I.e., if you care why
a normal iteration terminated, you can test after the fact. I.e., if all sequences
were the same length, the .rest() iterator will be empty. And if you don't care at
all about possible data, you can just try: it.rest().next() and catch StopIteration
to check.

BTW, is there any rule against passing information with StopIteration?

Regards,
Bengt Richter

 

[Szabolcs Nagy]

There are so many varieties of iterator that it's probably not workable

to alter the iterator API for all of the them.

i always wondered if it can be implemented:

there are iterators which has length:

i = iter([1,2,3])
len(i)

3

now isn't there a way to make this length inheritible?
eg. generators could have length in this case:

g = (x for x in [1,2,3])
# len(g) == len([1,2,3]) == 3

of course in special cases length would remain undefined:

f = (x for x in [1,2,3] if x>2)
# len(f) == ?

IMHO there are special cases when this is useful:
L=list(it)
here if it has length, then list creation can be more effective
(required memory is known in advance)

nsz

 

[Szabolcs Nagy]

There are so many varieties of iterator that it's probably not workable

to alter the iterator API for all of the them.

i always wondered if it can be implemented:

there are iterators which has length:

i = iter([1,2,3])
len(i)

3

now isn't there a way to make this length inheritible?
eg. generators could have length in this case:

g = (x for x in [1,2,3])
# len(g) == len([1,2,3]) == 3

of course in special cases length would remain undefined:

f = (x for x in [1,2,3] if x>2)
# len(f) == ?

IMHO there are special cases when this is useful:
L=list(it)
here if it has length, then list creation can be more effective
(required memory is known in advance)

nsz

 

[Paul Rubin]

there are iterators which has length:

i = iter([1,2,3])
len(i)

3

now isn't there a way to make this length inheritible?

I expect that's a __len__ method, which can be inherited.

eg. generators could have length in this case:

g = (x for x in [1,2,3])
# len(g) == len([1,2,3]) == 3

I dunno what happens with this now.

 

[Fredrik Lundh]

there are iterators which has length:

i = iter([1,2,3])
len(i)

3

that's a bug, which has been fixed in 2.5:

Python 2.5a0 (#5, Dec 14 2005, 22:28:52)
Type "help", "copyright", "credits" or "license" for more information.

i = iter([1, 2, 3])
len(i)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: len() of unsized object

</F>

 

[Raymond Hettinger]

[Raymond]

ISTM, these cures are worse than the disease ;-)

[Bengt]
Are you reacting to my turgidly rambling post, or to
...
('a', '1', 'A')
('b', '2', 'B')

Then after a backwards-compatible izip, if the iterator has
been bound, it can be used to continue, with sentinel sustitution:
...
('c', '<sentinel>', 'C')
('<sentinel>', '<sentinel>', 'D')

or optionally in sentinel substitution mode from the start:
...
('a', '1', 'A')
('b', '2', 'B')
('c', '<sentinel>', 'C')
('<sentinel>', '<sentinel>', 'D')

Usage-wise, this seems not too diseased to me, so I guess I want to make sure
this is what you were reacting to ;-)

There is an elegance to the approach; however, if some sort of fill-in
were needed, I'm more inclined to add a separate function than to
append a method to the izip object. The latter API presents a bit of a
learning/memory challenge, not because it is hard, but because it is
atypical.

A unique advantage for your API is that the loop can be run in two
phases, matched and unmatched. The question then turns to whether there
is a need for that option. So far, the three threads on the subject
have shown us to be starved for use cases for a single phase
izip_longest function, much less a two-pass version of the same.



Raymond