Making the case for "typed" lists/iterators in python

[Nathan Rice]

I realize this has been discussed in the past, I hope that I am
presenting a slightly different take on the subject that will prove
interesting. This is primarily motivated by my annoyance with using
comprehensions in certain circumstances.

Currently, if you want to perform successive transformations on the
elements of a list, a couple of options:

1. Successive comprehensions:

L2 = [X(e) for e in L1]
L3 = [Y(e) for e in L2]
L4 = [Z(e) for e in L3]
or
L2 = [e.X() for e in L1]

This gets the job done and gives you access to all the intermediate
values, but isn't very succinct, particularly if you are in the habit
of using informative identifiers.

2. One comprehension:

L2 = [Z(X(Y(e))) for e in L1]
or
L2 = [e.X().Y().Z() for e in L1]

This gets the job done, but doesn't give you access to all the
intermediate values, and tends to be pretty awful to read.

Having "typed" lists let you take preexisting string/int/etc methods
and expose them in a vectorized context and provides an easy way for
developers to support both vectors and scalars in a single function
(you could easily "fix" other people's functions dynamically to
support both). Additionally, "typed" lists/iterators will allow
improved code analysis and optimization. The PyPy people have already
stated that they are working on implementing different strategies for
lists composed of a single type, so clearly there is already community
movement in this direction.

Just compare the above examples to their type-aware counterparts:

L2 = X(L1)
L2 = L1.X()

L2 = Z(Y(X(L1)))
L2 = L1.X().Y().Z()

Also, this would provide a way to clean up stuff like:

"\n".join(l.capitalize() for l in my_string.split("\n"))

into:

my_string.split("\n").capitalize().join_this("\n")

Before anyone gets up in arms at the idea of statically typed python,
what I am suggesting here would be looser than that. Basically, I
believe it would be a good idea in instances where it is known that a
list of single type is going to be returned, to return a list subclass
(for example, StringList, IntegerList, etc). To avoid handcuffing
people with types, the standard list modification methods could be
hooked so that if an object of an incorrect type is placed in the
list, a warning is raised and the list converts to a generic object
list. The only stumbling block is that you can't use __class__ to
convert from stack types to heap types in CPython. My workaround for
this would be to have a factory that creates generic "List" classes,
modifying the bases to produce the correct behavior. Then, converting
from a typed list to a generic object list would just be a matter of
removing a member from the bases for a class. This of course
basically kills the ability to perform type specific list optimization
in CPython, but that isn't necessarily true for other implementations.
The additional type information would be preserved for code analysis
in any case. The case would be even simpler for generators and other
iterators, as you don't have to worry about mutation.

I'd like to hear people's thoughts on the subject. Currently we are
throwing away useful information in many cases that could be used for
code analysis, optimization and simpler interfaces. I believe that
"typed" lists that get "demoted" to normal lists with a warning on out
of type operations preserve this information while providing complete
backwards compatibility and freedom.

Nathan

 

[Roy Smith]

I'd like to hear people's thoughts on the subject. Currently we are
throwing away useful information in many cases that could be used for
code analysis, optimization and simpler interfaces.

Most of this was TLNR, but I will admit I often wish for a better way
to log intermediate values. For example, a common pattern in the code
I'm working with now is functions that end in:

return [Foo(x) for x in bunch_of_x_thingies]

When something goes amiss and I want to debug the problem, I often
transform that into:

temp = [Foo(x) for x in bunch_of_x_thingies]
logger.debug(temp)
return temp

It would be convenient to be able to get at and log the intermediate
value without having to pull it out to an explicit temporary.

 

[Arnaud Delobelle]

tee = lambda func,arg: (func(arg),arg)[1]

What a strange way to spell it!

def tee(func, arg):
func(arg)
return arg

 

[Mel Wilson]

It's no more strange than the way some people omit the u from colour.

Bonum Petronio Arbiteri, bonum mihi.

Mel.

 

[Terry Reedy]

I'm working with now is functions that end in:

return [Foo(x) for x in bunch_of_x_thingies]

When something goes amiss and I want to debug the problem, I often
transform that into:

temp = [Foo(x) for x in bunch_of_x_thingies]
logger.debug(temp)
return temp

It would be convenient to be able to get at and log the intermediate
value without having to pull it out to an explicit temporary.

Decorate the function with @logreturn and you do not have to touch the
function code. If the logging module does not now have such a decorator
predefined (I simply do not know), perhaps it should.

 

[Ulrich Eckhardt]

Am 16.12.2011 18:48, schrieb Nathan Rice:

I realize this has been discussed in the past, I hope that I am
presenting a slightly different take on the subject that will prove
interesting. This is primarily motivated by my annoyance with using
comprehensions in certain circumstances. [...]
Having "typed" lists let you take preexisting string/int/etc methods
and expose them in a vectorized context and provides an easy way for
developers to support both vectors and scalars in a single function
(you could easily "fix" other people's functions dynamically to
support both). Additionally, "typed" lists/iterators will allow
improved code analysis and optimization. The PyPy people have
already stated that they are working on implementing different
strategies for lists composed of a single type, so clearly there is
already community movement in this direction.

Just compare the above examples to their type-aware counterparts:

L2 = X(L1) L2 = L1.X()

L2 = Z(Y(X(L1))) L2 = L1.X().Y().Z()

Also, this would provide a way to clean up stuff like:

"\n".join(l.capitalize() for l in my_string.split("\n"))

into:

my_string.split("\n").capitalize().join_this("\n")

Before anyone gets up in arms at the idea of statically typed
python, what I am suggesting here would be looser than that.
Basically, I believe it would be a good idea in instances where it is
known that a list of single type is going to be returned, to return a
list subclass (for example, StringList, IntegerList, etc). To avoid
handcuffing people with types, the standard list modification methods
could be hooked so that if an object of an incorrect type is placed
in the list, a warning is raised and the list converts to a generic
object list. The only stumbling block is that you can't use
__class__ to convert from stack types to heap types in CPython. My
workaround for this would be to have a factory that creates generic
"List" classes, modifying the bases to produce the correct behavior.
Then, converting from a typed list to a generic object list would
just be a matter of removing a member from the bases for a class.
This of course basically kills the ability to perform type specific
list optimization in CPython, but that isn't necessarily true for
other implementations. The additional type information would be
preserved for code analysis in any case. The case would be even
simpler for generators and other iterators, as you don't have to
worry about mutation.

I'd like to hear people's thoughts on the subject. Currently we are
throwing away useful information in many cases that could be used
for code analysis, optimization and simpler interfaces.

I think there are two aspects to your idea:
1. collections that share a single type
2. accessing multiple elements via a common interface

Both are things that should be considered and I think both are useful in
some contexts. The former would provide additional guarantees, for
example, you could savely look up an attribute of the type only once
while iterating over the sequence and use it for all elements. Also, I
believe you could save some storage.

The second aspect would mean that you have a single function call that
targets multiple objects, which is syntactic sugar, but that's a good
thing. To some extent, this looks like a C++ valarray, see e.g. [1] and
[2] (note that I don't trust [1] and that [2] is perhaps a bit
outdated), in case you know C++ and want to draw some inspiration from this.

Anyway, I believe something like that would already be possible today,
which would give people something they could actually try out instead of
just musing about:

class ValarrayWrapper(object):
def __init__(self, elements):
self._elements = elements
def map(self, function):
tmp = [function(x) for x in self._elements]
return ValarrayWrapper(tmp)
def apply(self, function)
self._elements[:] = [function(x) for x in self._elements]

I could even imagine this to implement "generic" attribute lookup by
looking at the first element. If it contains the according attribute,
return a proxy that allows calls to member functions or property access,
depending on the type of the attribute.

I believe that "typed" lists that get "demoted" to normal lists with
a warning on out of type operations preserve this information while
providing complete backwards compatibility and freedom.

I don't think that a warning helps people write correct code, a
meaningful error does. Otherwise, with the same argument you could
convert a tuple on the fly to a list when someone tries to change an
element of it.


Cheers!

Uli



[1] http://www.cplusplus.com/reference/std/valarray/abs/
[2] http://drdobbs.com/184403620

 

[Nathan Rice]

I think there are two aspects to your idea:

1. collections that share a single type
2. accessing multiple elements via a common interface

You are correct, and I now regret posing them in a coupled manner.

Both are things that should be considered and I think both are useful in
some contexts. The former would provide additional guarantees, for example,
you could savely look up an attribute of the type only once while iterating
over the sequence and use it for all elements. Also, I believe you could
save some storage.

The second aspect would mean that you have a single function call that
targets multiple objects, which is syntactic sugar, but that's a good thing.
To some extent, this looks like a C++ valarray, see e.g. [1] and [2] (note
that I don't trust [1] and that [2] is perhaps a bit outdated), in case you
know C++ and want to draw some inspiration from this.

Anyway, I believe something like that would already be possible today, which
would give people something they could actually try out instead of just
musing about:

  class ValarrayWrapper(object):
      def __init__(self, elements):
          self._elements = elements
      def map(self, function):
          tmp = [function(x) for x in self._elements]
          return ValarrayWrapper(tmp)
      def apply(self, function)
          self._elements[:] = [function(x) for x in self._elements]

I could even imagine this to implement "generic" attribute lookup by looking
at the first element. If it contains the according attribute, return a proxy
that allows calls to member functions or property access, depending on the
type of the attribute.

Thank you for the references, I am always interested to see how other
languages solve problems.

I have received the "code please" comment repeatedly, I will have to
take some time after work today to deliver.

I don't think that a warning helps people write correct code, a meaningful
error does. Otherwise, with the same argument you could convert a tuple on
the fly to a list when someone tries to change an element of it.

I do agree errors are more normative than warnings. The problem with
an error in these circumstances is it will certainly break code
somewhere. Perhaps a warning that becomes an error at some point in
the future would be the prudent way to go.

Thanks!

Nathan