Iterate from 2nd element of a huge list

[Paulo da Silva]

Hi!

What is the best way to iterate thru a huge list having the 1st element
a different process? I.e.:

process1(mylist[0])
for el in mylist[1:]:
process2(el)

This way mylist is almost duplicated, isn't it?

Thanks.

 

[Cameron Simpson]

| What is the best way to iterate thru a huge list having the 1st element
| a different process? I.e.:
|
| process1(mylist[0])
| for el in mylist[1:]:
| process2(el)
|
| This way mylist is almost duplicated, isn't it?

Yep.

What about (untested):

process1(mylist[0])
for i in xrange(1,len(mylist)):
process2(mylist)

Cheers,
--
Cameron Simpson <[email protected]> DoD#743
http://www.cskk.ezoshosting.com/cs/

The truth is, I will not give myself the trouble to write sense long, for I
would as soon please fools as wise men; because fools are more numerous, and
every prudent man will go with the majority. - Hugh Henry Brackenridge

 

[Steven D'Aprano]

Hi!

What is the best way to iterate thru a huge list having the 1st element
a different process? I.e.:

process1(mylist[0])
for el in mylist[1:]:
process2(el)

This way mylist is almost duplicated, isn't it?

Yes. But don't be too concerned: what you consider a huge list and what
Python considers a huge list are unlikely to be the same. In my
experience, many people consider 10,000 items to be huge, but that's only
about 45K of memory. Copying it will be fast. On my laptop:

steve@runes:~$ python -m timeit -s "L = range(10000)" "L2 = L[1:]"
10000 loops, best of 3: 57.1 usec per loop

But if you have tens of millions of items, or think that you might
someday have to deal with tens of millions of items, here's an easy
technique to use:

it = iter(mylist)
process1(next(it)) # In Python 2.5, use it.next() instead.
for el in it:
process2(el)


No copying is performed.

For tiny lists, the iterator overhead will mean this is a smidgen slower,
but for tiny lists, who cares if it takes 2 nanoseconds instead of 1?

 

[Paulo da Silva]

Em 01-02-2012 01:39, Paulo da Silva escreveu:

Hi!

What is the best way to iterate thru a huge list having the 1st element
a different process? I.e.:

process1(mylist[0])
for el in mylist[1:]:
process2(el)

This way mylist is almost duplicated, isn't it?

Thanks.

I think iter is nice for what I need.
Thank you very much to all who responded.

 

[Paulo da Silva]

Em 01-02-2012 03:16, Paulo da Silva escreveu:

Em 01-02-2012 01:39, Paulo da Silva escreveu:

Hi!

What is the best way to iterate thru a huge list having the 1st element
a different process? I.e.:

process1(mylist[0])
for el in mylist[1:]:
process2(el)

This way mylist is almost duplicated, isn't it?

Thanks.

I think iter is nice for what I need.
Thank you very much to all who responded.

BTW, iter seems faster than iterating thru mylist[1:]!

 

[Cameron Simpson]

| Em 01-02-2012 03:16, Paulo da Silva escreveu:
| > I think iter is nice for what I need.
| > Thank you very much to all who responded.
|
| BTW, iter seems faster than iterating thru mylist[1:]!

I would hope the difference can be attributed to the cost of copying
mylist[1:]. Do your timings suggest this? (Remembering also that for
most benchmarking you need to run things many times unless the effect
is quite large).

Cheers,
--
Cameron Simpson <[email protected]> DoD#743
http://www.cskk.ezoshosting.com/cs/

Any company large enough to have a research lab
is large enough not to listen to it. - Alan Kay

 

[Arnaud Delobelle]

Em 01-02-2012 01:39, Paulo da Silva escreveu:

Hi!

What is the best way to iterate thru a huge list having the 1st element
a different process? I.e.:

process1(mylist[0])
for el in mylist[1:]:
      process2(el)

This way mylist is almost duplicated, isn't it?

Thanks.

I think iter is nice for what I need.
Thank you very much to all who responded.

Nobody mentioned itertools.islice, which can be handy, especially if
you weren't interested in the first element of the list:

from itertools import islice:

for el in islice(mylist, 1):
process2(el)

 

[Peter Otten]

Nobody mentioned itertools.islice, which can be handy, especially if
you weren't interested in the first element of the list:

Also, skipping two or seven or ... items is just as easy.
The example should be

from itertools import islice:

for el in islice(mylist, 1, None):

 

[Paul Rubin]

process1(mylist[0])
for el in mylist[1:]:
process2(el)

This way mylist is almost duplicated, isn't it?

I think it's cleanest to use itertools.islice to get the big sublist
(not tested):

from itertools import islice

process1 (mylist[0])
for el in islice(mylist, 1, None):
process2 (el)

The islice has a small, constant amount of storage overhead instead of
duplicating almost the whole list.

 

[Arnaud Delobelle]

Arnaud Delobelle wrote:
The example should be


for el in islice(mylist, 1, None):

Oops!

 

[Stefan Behnel]

Paul Rubin, 01.02.2012 10:25:

process1(mylist[0])
for el in mylist[1:]:
process2(el)

This way mylist is almost duplicated, isn't it?

I think it's cleanest to use itertools.islice to get the big sublist
(not tested):

from itertools import islice

process1 (mylist[0])
for el in islice(mylist, 1, None):
process2 (el)

The islice has a small, constant amount of storage overhead instead of
duplicating almost the whole list.

It also has a tiny runtime overhead, though. So, if your code is totally
performance critical and you really just want to strip off the first
element and then run through all the rest, it may still be better to go the
iter() + next() route.

python3.3 -m timeit -s 'l=list(range(100000))' \
'it = iter(l); next(it); all(it)'
1000 loops, best of 3: 935 usec per loop

python3.3 -m timeit -s 'l=list(range(100000))' \
-s 'from itertools import islice' \
'all(islice(l, 1, None))'
1000 loops, best of 3: 1.63 msec per loop

Stefan

 

[Paulo da Silva]

Em 01-02-2012 04:55, Cameron Simpson escreveu:

On 01Feb2012 03:34, Paulo da Silva <[email protected]> wrote:

| BTW, iter seems faster than iterating thru mylist[1:]!

I would hope the difference can be attributed to the cost of copying
mylist[1:].

I don't think so. I tried several times and the differences were almost
always consistent.

I put mylist1=mylist[1:] outside the time control. iter still seems a
little bit faster. Running both programs several times (10000000
elements list) I only got iter being slower once!

But, of course, most of the difference comes from the copy.

 

[Steven D'Aprano]

Em 01-02-2012 04:55, Cameron Simpson escreveu:

On 01Feb2012 03:34, Paulo da Silva <[email protected]>
wrote:

| BTW, iter seems faster than iterating thru mylist[1:]!

I would hope the difference can be attributed to the cost of copying
mylist[1:].

I don't think so. I tried several times and the differences were almost
always consistent.

Yes, actually iterating over a list-iterator appears to be trivially
faster (although this may not apply to arbitrary iterators):

steve@runes:~$ python -m timeit -s "L=range(10000)" "for x in L: pass"
1000 loops, best of 3: 280 usec per loop
steve@runes:~$ python -m timeit -s "L=range(10000)" "for x in iter(L):
pass"
1000 loops, best of 3: 274 usec per loop


The difference of 6 microseconds would be lost in the noise if the loops
actually did something useful.

Also keep in mind that for tiny lists, the overhead of creating the
iterator is probably much greater than the time of iterating over the
list:

steve@runes:~$ python -m timeit -s "L=range(3)" "for x in L: pass"
1000000 loops, best of 3: 0.238 usec per loop
steve@runes:~$ python -m timeit -s "L=range(3)" "for x in iter(L): pass"
1000000 loops, best of 3: 0.393 usec per loop

But of course the difference is only relatively significant, in absolute
terms nobody is going to notice an extra 0.1 or 0.2 microseconds.