more fun with iterators (mux, demux)

[Neal Becker]

I'm trying to make a multiplexor and demultiplexor, using generators. The
multiplexor will multiplex N sequences -> 1 sequence (assume equal length).
The demultiplexor will do the inverse.

The mux seems easy enough:

-----------------------
def mux (*ranges):
iterables = [iter (r) for r in ranges]
while (True):
for i in (iterables):
yield i.next()

def test_mux ():
a = xrange (10)
b = xrange (10,20)
return mux (a, b)

x = test_mux()
print [e for e in x]
------------------------

The demux has me stumped. The demux should return a tuple of N generators.
The signature should be:

def demux (sequence, N):

Not sure about this one.

 

[Steven D'Aprano]

I'm trying to make a multiplexor and demultiplexor, using generators.
The multiplexor will multiplex N sequences -> 1 sequence (assume equal
length). The demultiplexor will do the inverse.

The mux seems easy enough:

-----------------------
def mux (*ranges):
iterables = [iter (r) for r in ranges] while (True):
for i in (iterables):
yield i.next()

This is like a zip, and can be re-written using itertools.izip.

def mux(*iterables):
for i in itertools.izip(*iterables):
for item in i:
yield item


The demuxer can't be an iterator, since it needs to run through the
entire collection.

def demux(it, n):
collectors = [[] for i in xrange(n)]
i = 0
for item in it:
collectors.append(item)
i = (i+1) % n
return tuple([iter(x) for x in collectors])

 

[Miles]

I'm trying to make a multiplexor and demultiplexor, using generators.
The multiplexor will multiplex N sequences -> 1 sequence  (assume equal
length). The demultiplexor will do the inverse.

The mux seems easy enough:

-----------------------
def mux (*ranges):
    iterables = [iter (r) for r in ranges] while (True):
        for i in (iterables):
            yield i.next()

This is like a zip, and can be re-written using itertools.izip.

def mux(*iterables):
   for i in itertools.izip(*iterables):
       for item in i:
           yield item

In Python 2.6, you could also do this:

def mux(*iterables):
return itertools.chain.from_iterable(itertools.izip(*iterables))

-Miles

 

[Neal Becker]

The demuxer can't be an iterator, since it needs to run through the
entire collection.

Then your demuxer obviously cannot handle infinite sequences.

def demux(it, n):
collectors = [[] for i in xrange(n)]
i = 0
for item in it:
collectors.append(item)
i = (i+1) % n
return tuple([iter(x) for x in collectors])

But this one can:

from collections import deque
from itertools import cycle, izip, count, islice

def mux(*iterables):
for it in izip(*iterables):
for item in it:
yield item

def demux(seq,n):
it = iter(seq)
Q = [deque() for i in xrange(n)]
CQ = cycle(Q)
def gen(D):
for x,C in izip(it,CQ):
C.appendleft(x)
while D:
yield D.pop()
while D:
yield D.pop()
return map(gen,Q)

def test():
a = count(10)
b = count(20)
c = count (30)
x = demux(mux(a,b,c),3)
for e in x:
print list(islice(e,0,10))

if __name__=='__main__':
test()

P.

What was wrong with this one?

def demux(iterable, n):
return tuple(islice(it, i, None, n) for (i, it) in
enumerate(tee(iterable, n)))

 

[pataphor]

What was wrong with this one?

def demux(iterable, n):
return tuple(islice(it, i, None, n) for (i, it) in
enumerate(tee(iterable, n)))

Nothing much, I only noticed after posting that this one handles
infinite sequences too. For smallish values of n it is acceptable.

P.

 

[Miles]

Nothing much, I only noticed after posting that this one handles
infinite sequences too. For smallish values of n it is acceptable.

I assume that "smallish values of n" refers to the fact that
itertools.tee places items into every generator's internal deque,
which islice then skips over, whereas your version places items only
into the deque of the generator that needs it. However, for small n,
the tee-based solution has the advantage of having most of the work
done in C instead of in Python generator functions; in my limited
benchmarking, the point where your version becomes faster is somewhere
around n=65.

-Miles

 

[Raymond Hettinger]

[Miles]

I assume that "smallish values of n" refers to the fact that
itertools.tee places items into every generator's internal deque,
which islice then skips over, whereas your version places items only
into the deque of the generator that needs it.

The pure python equivalent listed in the docs uses separate
deques for each subiterator, but the actual C implementation
of itertools.tee() only has one internal deque that is shared
by all of the sub-iterators.


Raymond