Split a list into two parts based on a filter?

[rusi]

[Of course I would prefer a 3-liner where the body of the for is
indented ]
Is this an aside comprehension?

Eh?

I know they always say "don't explain the joke", but I'll have a shot
at it. It's somewhat like an autopsy though - you find out why it
ticks, but in the process, you prove that it's no longer ticking...

Ok

Missed that my aside looked like a comprehension. Or rather did not
comprehend!

 

[Peter Otten]

I have to tell you this is the coolest thing I've read today. I'd
never have thought of that.

 

[Chris Angelico]

11.06.13 01:50, Chris Angelico напиÑав(ла):


new_songs = [s for s in songs if s.is_new()]
old_songs = [s for s in songs if not s.is_new()]


Hmm. Would this serve?

old_songs = songs[:]
new_songs = [songs.remove(s) or s for s in songs if s.is_new()]

I think you meant old_songs.remove(s).

Ah, yes, editing fail. I started by mutating the original list, then
thought "Oh, better to work with a copy"... and forgot to complete the
edit.

Which isn't significant if len(songs) is low. We weren't told the
relative costs - is the is_new call ridiculously expensive? Everything
affects algorithmic choice.

But is it correct? In the general case, no:

numbers = [1, 1.0, 2.0, 2]
ints = numbers[:]
floats = [ints.remove(n) or n for n in numbers if isinstance(n, float)]
floats [1.0, 2.0]
ints

[1.0, 2] # hmm

Sure, but the implication of the original is that they're uniquely
identifiable. Anyway, it wasn't meant to be absolutely perfect, just
another notion getting thrown out there... and then thrown out

ChrisA

 

[Roel Schroeven]

Peter Otten schreef:

You could do something like:

new_songs, old_songs = [], []
[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

But I'm not sure that that's any better than the long version.

This is so beautiful!

It makes me cringe.

This code does spurious work to turn what is naturally written as a for loop
into a list comprehension that is thrown away immediately.

I agree. I see two problems with it: it's not very readable or pythonic,
and it creates a list that is not needed and is immediately thrown away.

I wrote that code just to see if I could make it work that way. In real
code I would never write it that way, let's make that clear


--
"People almost invariably arrive at their beliefs not on the basis of
proof but on the basis of what they find attractive."
-- Pascal Blaise

(e-mail address removed)

 

[Roy Smith]

11.06.13 01:50, Chris Angelico напиÑав(ла):

new_songs = [s for s in songs if s.is_new()]
old_songs = [s for s in songs if not s.is_new()]


Hmm. Would this serve?

old_songs = songs[:]
new_songs = [songs.remove(s) or s for s in songs if s.is_new()]

O(len(songs)**2) complexity.

If I didn't want to make two passes over songs, I probably don't want
something that's O(len(songs)^2)

Which isn't significant if len(songs) is low. We weren't told the
relative costs - is the is_new call ridiculously expensive? Everything
affects algorithmic choice.

Assume is_new() is cheap. It's essentially:

return (datetime.utcnow() - self.create_time) < [[a pre-defined timedelta]]

 

[Roy Smith]

11.06.13 07:11, Roy Smith напиÑав(ла):

new_songs, old_songs = [], []
[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

Thanks kind of neat, thanks.

I'm trying to figure out what list gets created and discarded. I think
it's [None] * len(songs).

It is the same as your klunky code, but consumes more memory.

Well, continuing down this somewhat bizarre path:

new_songs, old_songs = [], []
itertools.takewhile(
lambda x: True,
(new_songs if s.is_new() else old_songs).append(s) for s in songs)
)

I'm not sure I got the syntax exactly right, but the idea is anything
that will iterate over a generator expression. That at least gets rid
of the memory requirement to hold the throw-away list

 

[alex23]

new_songs, old_songs = [], []
[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

This is so beautiful!

No, it's actually pretty terrible. It creates a list in order to
populate _two other lists_ and then throws away the one made by the
comprehension. There's nothing ugly about multiline for-loops.

 

[Phil Connell]

11.06.13 07:11, Roy Smith напиÑав(ла):

new_songs, old_songs = [], []
[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

Thanks kind of neat, thanks.

I'm trying to figure out what list gets created and discarded. I think
it's [None] * len(songs).

It is the same as your klunky code, but consumes more memory.

Well, continuing down this somewhat bizarre path:

new_songs, old_songs = [], []
itertools.takewhile(
lambda x: True,
(new_songs if s.is_new() else old_songs).append(s) for s in songs)
)

I'm not sure I got the syntax exactly right, but the idea is anything
that will iterate over a generator expression. That at least gets rid
of the memory requirement to hold the throw-away list

You could equivalently pass the generator to deque() with maxlen=0 - this
consumes the iterator with constant memory usage.

We are of course firmly in the twilight zone at this point (although this
can be a useful technique in general).

Cheers,
Phil

 

[Roy Smith]

Well, continuing down this somewhat bizarre path:

new_songs, old_songs = [], []
itertools.takewhile(
lambda x: True,
(new_songs if s.is_new() else old_songs).append(s) for s in songs)
)

I'm not sure I got the syntax exactly right, but the idea is anything
that will iterate over a generator expression. That at least gets rid
of the memory requirement to hold the throw-away list

You could equivalently pass the generator to deque() with maxlen=0 - this
consumes the iterator with constant memory usage.

We are of course firmly in the twilight zone at this point (although this
can be a useful technique in general).[/QUOTE]

We've been in the twilight zone for a while. That's when the fun
starts. But, somewhat more seriously, I wonder what, exactly, it is
that freaks people out about:

[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

Clearly, it's not the fact that it build and immediately discards a
list, because that concern is addressed with the generator hack, and I
think everybody (myself included) agrees that's just horrible.

Or, is it the use of the conditional to create the target for append()?
Would people be as horrified if I wrote:

for s in songs:
(new_songs if s.is_new() else old_songs).append(s)

or even:

for s in songs:
the_right_list = new_songs if s.is_new() else old_songs
the_right_list.append(s)

 

[Fábio Santos]

Well, continuing down this somewhat bizarre path:

new_songs, old_songs = [], []
itertools.takewhile(
lambda x: True,
(new_songs if s.is_new() else old_songs).append(s) for s in songs)
)

I'm not sure I got the syntax exactly right, but the idea is anything
that will iterate over a generator expression. That at least gets rid
of the memory requirement to hold the throw-away list

You could equivalently pass the generator to deque() with maxlen=0 - this
consumes the iterator with constant memory usage.

We are of course firmly in the twilight zone at this point (although this
can be a useful technique in general).

We've been in the twilight zone for a while. That's when the fun
starts. But, somewhat more seriously, I wonder what, exactly, it is
that freaks people out about:

[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

Clearly, it's not the fact that it build and immediately discards a
list, because that concern is addressed with the generator hack, and I
think everybody (myself included) agrees that's just horrible.[/QUOTE]

I think someone has already said that the problem is that you're creating
the list comprehension just for the side effects.

Or, is it the use of the conditional to create the target for append()?

I particularly liked that part.

 

[Jussi Piitulainen]

We've been in the twilight zone for a while. That's when the fun
starts. But, somewhat more seriously, I wonder what, exactly, it is
that freaks people out about:

[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

Clearly, it's not the fact that it build and immediately discards a
list, because that concern is addressed with the generator hack, and
I think everybody (myself included) agrees that's just horrible.

I expect e(s) in [e(s) for s in songs] to be an expression that is
evaluated for its value and not for an effect. The comprehension
should stand for a list of those values. The one above violates this
expectation.

Or, is it the use of the conditional to create the target for append()?
Would people be as horrified if I wrote:

for s in songs:
(new_songs if s.is_new() else old_songs).append(s)

or even:

for s in songs:
the_right_list = new_songs if s.is_new() else old_songs
the_right_list.append(s)

These are fine.

 

[Serhiy Storchaka]

12.06.13 09:32, Phil Connell напиÑав(ла):

Well, continuing down this somewhat bizarre path:

new_songs, old_songs = [], []
itertools.takewhile(
lambda x: True,
(new_songs if s.is_new() else old_songs).append(s) for s in songs)
)

I'm not sure I got the syntax exactly right, but the idea is anything
that will iterate over a generator expression. That at least gets rid
of the memory requirement to hold the throw-away list

You could equivalently pass the generator to deque() with maxlen=0 -
this consumes the iterator with constant memory usage.

any((new_songs if s.is_new() else old_songs).append(s) for s in songs)

 

[Fábio Santos]

Why is there no builtin to consume a generator? I find that odd.

12.06.13 09:32, Phil Connell напиÑав(ла):

Well, continuing down this somewhat bizarre path:

new_songs, old_songs = [], []
itertools.takewhile(
lambda x: True,
(new_songs if s.is_new() else old_songs).append(s) for s in songs)
)

I'm not sure I got the syntax exactly right, but the idea is anything
that will iterate over a generator expression. That at least gets rid
of the memory requirement to hold the throw-away list

You could equivalently pass the generator to deque() with maxlen=0 -
this consumes the iterator with constant memory usage.

any((new_songs if s.is_new() else old_songs).append(s) for s in songs)

 

[Terry Reedy]

starts. But, somewhat more seriously, I wonder what, exactly, it is
that freaks people out about:

[(new_songs if s.is_new() else old_songs).append(s) for s in songs]

Clearly, it's not the fact that it build and immediately discards a
list, because that concern is addressed with the generator hack, and I
think everybody (myself included) agrees that's just horrible.

It is an example of comprehension abuse. Comprehensions express and
condense a stylized pattern of creating collections from another
collection or collections, possibly filtered. They were not mean to
replace for statements and turn Python into an fp languages. Indeed,
they do replace and expand upon the fp map function. Python for loops
are not evil.

Or, is it the use of the conditional to create the target for append()?
Would people be as horrified if I wrote:

for s in songs:
(new_songs if s.is_new() else old_songs).append(s)

No. That succinctly expresses and implements the idea 'append each song
to one of two lists.

 

[Terry Reedy]

Why is there no builtin to consume a generator? I find that odd.

There are several builtins than consume generators -- and do something
useful with the yielded objects. What you mean is "Why is there no
builtin to uselessly consume a generator?" The question almost answers
itself. A generator generates objects to be used.

any((new_songs if s.is_new() else old_songs).append(s) for s in songs)

[/QUOTE]

The problem here is that the generator generates and yields an unwanted
sequence of None (references) from the append operations. The proper
loop statement

for s in songs:
(new_songs if s.is_new() else old_songs).append(s)

simply ignores the None return of the appends. Since it does not yield
None over and over, no extra code is needed to ignore what should be
ignored in the first place.

 

[Oscar Benjamin]

The proper loop statement

for s in songs:
(new_songs if s.is_new() else old_songs).append(s)

I think I would just end up rewriting this as

for s in songs:
if s.is_new():
new_songs.append(s)
else:
old_songs.append(s)

but then we're back where we started. I don't think any of the
solutions posted in this thread have been better than this. If you
want to make this a nice one-liner then just put this code in a
function.


Oscar