Using iterators to write in the structure being iterated through?

[Pierre Thibault]

Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

template <class H>
class MyClass
{
public:

MyClass(H& o1, H& o2) : object1(o1), object2(o2) {}

void compute();

private:
H& object1;
H& object2;

};

template <class H>
void MyClass::compute()
{
typedef typename H::iterator I;

I o1_begin = object1.begin();
I o2_begin = object2.begin();
I o1_end = object1.end();

for(I io1 = o1_begin, io2 = o2_begin; io1 != o1_end; ++io1, ++io2)
{
// Do something with *io1 and *io2, for instance:
// *io1 += *io2;
}
}

This is all nice: any object having a forward iterator works in there.

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre

 

[Chris Lambacher]

I think you are going to need to provide some python minimal code as an
example of what is not working for you.

-Chris

 

[Peter Otten]

Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

template <class H>
class MyClass
{
public:

MyClass(H& o1, H& o2) : object1(o1), object2(o2) {}

void compute();

private:
H& object1;
H& object2;

};

template <class H>
void MyClass::compute()
{
typedef typename H::iterator I;

I o1_begin = object1.begin();
I o2_begin = object2.begin();
I o1_end = object1.end();

for(I io1 = o1_begin, io2 = o2_begin; io1 != o1_end; ++io1, ++io2)
{
// Do something with *io1 and *io2, for instance:
// *io1 += *io2;
}
}

This is all nice: any object having a forward iterator works in there.

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

You need a temporary variable (out in the example below):

accus = [[] for i in range(3)]
ins = ("abc" for i in range(3))
outs = iter(accus)
while 1:

.... out = outs.next()
.... out += ins.next()
....
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
StopIteration

In idiomatic Python that becomes

accus = [[] for i in range(3)]
ins = ("abc" for i in range(3))
outs = iter(accus)
from itertools import izip
for out, in_ in izip(outs, ins):

.... out += in_
....[['a', 'b', 'c'], ['a', 'b', 'c'], ['a', 'b', 'c']]


Peter

 

[Pierre Thibault]

Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

template <class H>
class MyClass
{
public:

MyClass(H& o1, H& o2) : object1(o1), object2(o2) {}

void compute();

private:
H& object1;
H& object2;

};

template <class H>
void MyClass::compute()
{
typedef typename H::iterator I;

I o1_begin = object1.begin();
I o2_begin = object2.begin();
I o1_end = object1.end();

for(I io1 = o1_begin, io2 = o2_begin; io1 != o1_end; ++io1, ++io2)
{
// Do something with *io1 and *io2, for instance:
// *io1 += *io2;
}
}

This is all nice: any object having a forward iterator works in there.

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

You need a temporary variable (out in the example below):

accus = [[] for i in range(3)]
ins = ("abc" for i in range(3))
outs = iter(accus)
while 1:

... out = outs.next()
... out += ins.next()
...
Traceback (most recent call last):
File "<stdin>", line 2, in <module>
StopIteration

In idiomatic Python that becomes

accus = [[] for i in range(3)]
ins = ("abc" for i in range(3))
outs = iter(accus)
from itertools import izip
for out, in_ in izip(outs, ins):

... out += in_
...[['a', 'b', 'c'], ['a', 'b', 'c'], ['a', 'b', 'c']]


Peter

Hum, this example seems like a special case not really appropriate for my
needs. Let me make my problem a little more precise. The objects I'll want
to iterate through will always contain some floats. Very often, I guess
they will be numpy.ndarray instances, but they can be something else as
well. For instance, I will have to deal with arrays having internal
symmetries (crystallographic datasets). The most efficient thing to do
with those is save only the unique values and keep track of the symmetry
operations needed to extract other values.

A 1d examples: I have a class which represents a unit cell
which has the additional mirror symmetry in the middle of the cell. Say
C is an instance of this class representing data on a 50-long array. Only
25 datum will be stored in C, and the class takes care of giving the value
of C[0] if C[49] is asked, C[1] for C[48], and so on. Since crystals are
periodic, the translational symmetry can also be managed (C[-1] == C[49]
== C[0], C[-2] == C[48] == C[1]...). In any case, the user of this object
need not know how the data is stored: for him, C is just a funny
array-like object defined from -infinity to +infinity on a 1-D lattice.

Now, I want to do simple math operations on the data in C. Doing a loop
from 0 to 49 would loop twice through the actual data. In this
context, an iterator is perfect since it can take care internally of going
through the data only once, so it would be great to access _AND MODIFY_
data over which I iterate.

I now realize I don't know how to do that with numpy.ndarray either. It
looks like the only way I can modify the content of an array is by using
the [] notation (that is, in random access). For instance, the
following will not change the state of a:

import numpy
a = numpy.array([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.]])

for c in a.flat
c += 2.

Of course, I know that a += .2 would be alright, but this is not general
enough for more complicated classes.

So I guess my problem is worst than I thought.

Thanks anyway for your help!

Pierre

 

[Peter Otten]

Hum, this example seems like a special case not really appropriate for my
needs. Let me make my problem a little more precise. The objects I'll want
to iterate through will always contain some floats. Very often, I guess
they will be numpy.ndarray instances, but they can be something else as
well. For instance, I will have to deal with arrays having internal
symmetries (crystallographic datasets). The most efficient thing to do
with those is save only the unique values and keep track of the symmetry
operations needed to extract other values.

A 1d examples: I have a class which represents a unit cell
which has the additional mirror symmetry in the middle of the cell. Say
C is an instance of this class representing data on a 50-long array. Only
25 datum will be stored in C, and the class takes care of giving the value
of C[0] if C[49] is asked, C[1] for C[48], and so on. Since crystals are
periodic, the translational symmetry can also be managed (C[-1] == C[49]
== C[0], C[-2] == C[48] == C[1]...). In any case, the user of this object
need not know how the data is stored: for him, C is just a funny
array-like object defined from -infinity to +infinity on a 1-D lattice.

Now, I want to do simple math operations on the data in C. Doing a loop
from 0 to 49 would loop twice through the actual data. In this
context, an iterator is perfect since it can take care internally of going
through the data only once, so it would be great to access _AND MODIFY_
data over which I iterate.

That would not only be nice but crucial to the integrity of your data. You'd
have to explicitly forbid operations on individual cells as applying an
operation to a single cell would change two or more cells, depending of the
kind of symmetry.

I now realize I don't know how to do that with numpy.ndarray either. It
looks like the only way I can modify the content of an array is by using
the [] notation (that is, in random access). For instance, the
following will not change the state of a:

import numpy
a = numpy.array([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.]])

for c in a.flat
c += 2.

Of course, I know that a += .2 would be alright, but this is not general
enough for more complicated classes.

So I guess my problem is worst than I thought.

Indeed, but I don't think it would be easier in C++...

Peter

 

[bearophileHUGS]

Pierre Thibault, some people here surely know enough Python (and C++)
to solve your problem, but often the problem is understanding the
problem. I have understood your problem just partially, so the
following are just ideas.

First of all I suggest you to use a normal Python list to keep the
data, and later modify the code to use an array if it is too much slow
or if you need arrays to do some special things, like plotting, etc.
Developing using normal Python lists is a bit simpler.

With python you can redefine the __getitem__ and __setitem__ of an
object according to its crystal symmetries. Then you can define some
method like iterkeys that yields (with yield) the only indexes that the
symmetry allows to be set.

Then you can iterate on those indexes and update the values in the list
attribute.

Bye,
bearophile

 

[Paddy]

Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre

Pierre,
You should be able to write
io1.next().param += io2.next().param
If iter(object1) and iter(object2) both return classes or instances
with the appropriate parameter.
Here is what I was thinking of:


class ParamHolder(object):
def __init__(self, n):
self.param = n

class C1(object):
def __init__(self,m):
self.value = [ParamHolder(n) for n in range(m)]
def __getitem__(self, p):
return self.value[p]

obj1 = C1(5)
obj2 = C1(5)

io1 = iter(obj1)
io2 = iter(obj2)

print "obj1 pre loop",[r.param for r in obj1.value]

try:
while 1:
io1.next().param += io2.next().param
except StopIteration:
pass

print "obj1 post loop",[r.param for r in obj1.value]

- Paddy.

 

[Paddy]

Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre

Pierre,
You should be able to write
io1.next().param += io2.next().param
If iter(object1) and iter(object2) both return classes or instances
with the appropriate parameter.
Here is what I was thinking of:


class ParamHolder(object):
def __init__(self, n):
self.param = n

class C1(object):
def __init__(self,m):
self.value = [ParamHolder(n) for n in range(m)]
def __getitem__(self, p):
return self.value[p]

obj1 = C1(5)
obj2 = C1(5)

io1 = iter(obj1)
io2 = iter(obj2)

print "obj1 pre loop",[r.param for r in obj1.value]

try:
while 1:
io1.next().param += io2.next().param
except StopIteration:
pass

print "obj1 post loop",[r.param for r in obj1.value]

- Paddy.

I don't like the try/except code and would write something like the
following:
.... x.param += y.param
....

print "obj1 post for loop",[r.param for r in obj1.value] obj1 post for loop [0, 2, 4, 6, 8]

- Paddy.

 

[Pierre Thibault]

Hello!

I am currently trying to port a C++ code to python, and I think I am stuck
because of the very different behavior of STL iterators vs python
iterators. What I need to do is a simple arithmetic operations on objects
I don't know. In C++, the method doing that was a template, and all that
was required is that the template class has an iterator conforming to the
STL forward iterator definition. Then, the class would look like:

Then I discovered python and wanted to use all its goodies. I thought it
would be easy to do the same thing but I can't: the iterator mechanism is
read-only, right? So it does no make sense to write:

io1 = iter(object1)
io2 = iter(object2)

try:
while 1:
io1.next() += io2.next()
except StopIteration:
pass

That won't work:
SyntaxError: can't assign to function call

Here is my question: how could I do that and retain enough generallity?

Thanks!

Pierre

Pierre,
You should be able to write
io1.next().param += io2.next().param
If iter(object1) and iter(object2) both return classes or instances
with the appropriate parameter.
Here is what I was thinking of:


class ParamHolder(object):
def __init__(self, n):
self.param = n

class C1(object):
def __init__(self,m):
self.value = [ParamHolder(n) for n in range(m)]
def __getitem__(self, p):
return self.value[p]

obj1 = C1(5)
obj2 = C1(5)

io1 = iter(obj1)
io2 = iter(obj2)

print "obj1 pre loop",[r.param for r in obj1.value]

try:
while 1:
io1.next().param += io2.next().param
except StopIteration:
pass

print "obj1 post loop",[r.param for r in obj1.value]

- Paddy.

I don't like the try/except code and would write something like the
following:
... x.param += y.param
...

print "obj1 post for loop",[r.param for r in obj1.value] obj1 post for loop [0, 2, 4, 6, 8]

- Paddy.

Thanks Paddy,

This looks like the closest thing to what I wanted, though the need for
this "param" makes it not very general to my taste. Besides, this method
would not work with ndarrays anyways.

Thanks again for taking the time to answer,

Pierre

 

[Pierre Thibault]

Hum, this example seems like a special case not really appropriate for my
needs. Let me make my problem a little more precise. The objects I'll want
to iterate through will always contain some floats. Very often, I guess
they will be numpy.ndarray instances, but they can be something else as
well. For instance, I will have to deal with arrays having internal
symmetries (crystallographic datasets). The most efficient thing to do
with those is save only the unique values and keep track of the symmetry
operations needed to extract other values.

A 1d examples: I have a class which represents a unit cell
which has the additional mirror symmetry in the middle of the cell. Say
C is an instance of this class representing data on a 50-long array. Only
25 datum will be stored in C, and the class takes care of giving the value
of C[0] if C[49] is asked, C[1] for C[48], and so on. Since crystals are
periodic, the translational symmetry can also be managed (C[-1] == C[49]
== C[0], C[-2] == C[48] == C[1]...). In any case, the user of this object
need not know how the data is stored: for him, C is just a funny
array-like object defined from -infinity to +infinity on a 1-D lattice.

Now, I want to do simple math operations on the data in C. Doing a loop
from 0 to 49 would loop twice through the actual data. In this
context, an iterator is perfect since it can take care internally of going
through the data only once, so it would be great to access _AND MODIFY_
data over which I iterate.

That would not only be nice but crucial to the integrity of your data. You'd
have to explicitly forbid operations on individual cells as applying an
operation to a single cell would change two or more cells, depending of the
kind of symmetry.

I now realize I don't know how to do that with numpy.ndarray either. It
looks like the only way I can modify the content of an array is by using
the [] notation (that is, in random access). For instance, the
following will not change the state of a:

import numpy
a = numpy.array([[1.,2.,3.],[4.,5.,6.],[7.,8.,9.]])

for c in a.flat
c += 2.

Of course, I know that a += .2 would be alright, but this is not general
enough for more complicated classes.

So I guess my problem is worst than I thought.

Indeed, but I don't think it would be easier in C++...

Peter

Well, no, C++ is better in this case because the thing you iterate over is
a generalized pointer, which can be dereferenced to get access to the
memory. So in other word, iterator.next() (in python) is a copy, while
*iterator (in C++) is a reference.

Thanks anyway for your comments!

Pierre

 

[Pierre Barbier de Reuille]

Pierre Thibault wrote:

[...]

Now, I want to do simple math operations on the data in C. Doing a loop
from 0 to 49 would loop twice through the actual data. In this
context, an iterator is perfect since it can take care internally of going
through the data only once, so it would be great to access _AND MODIFY_
data over which I iterate.

The biggest problem I think is that numbers are immutable in Python.
Thus you cannot modify a number if you have a variable on it! All you
can do is modify the content of the variable.

IMHO, the way to go is to iterate on the indices and not on the values
(not much slower in Python). For example you can do something like:

lst1 = [1,2,3,4]
lst2 = [4,3,4,5]

for i in xrange(len(lst1)):
lst1 += lst2

If you *really* want to iterate (for example to support something else
that lists), then you will need to create a new structure like that:

from itertool import izip

lst1 = MyList1()
lst2 = MyList2()

result = [0]*len(lst1) # If possible

for i,(e1,e2) in enumerate(izip(lst1, lst2)):
result = e1+e2

lst1[:] = result # If you really need it

An intermediate possibility would be (if lst1 is indexable):

for i,e2 in enumerate(lst2):
lst1 += e2

But it all depend on the kind of structure you get.

Pierre