C replacement for Queue module

[Jonathan Ellis]

I'm working on an application that makes heavy use of Queue objects in
a multithreaded environment.

By "heavy" I mean "millions of calls to put and get, constituting ~20%
of the app's run time." The profiler thinks that a significant amount
of time is spent in this code -- not just a consumer waiting for a
producer, but actual _empty, notify calls, etc.

Would it be worth the time to write a CQueue module with pthread_cond
instead of Python Condition objects, etc? I don't really have a gut
feeling for how much bang-for-the-loc C optimization might give: twice
as fast? 5x as fast?

thanks,

-Jonathan

 

[Jason Lai]

How about Python 2.4's collections.deque class? Supposedly it's
thread-safe, and it's implemented in C.

- Jason

 

[jepler]

does collections.deque have a blocking popleft()? If not, it's not very
suitable to replace Queue.Queue.

Jeff

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)

iD8DBQFDWbH1Jd01MZaTXX0RAm4yAJ4tQtff8/SjtnOkR0Zlrr5LNnSBFACaAyPr
L714IVqIIOBPy2UsTLz0krI=
=bCJe
-----END PGP SIGNATURE-----

 

[Jonathan Ellis]

does collections.deque have a blocking popleft()? If not, it's not very
suitable to replace Queue.Queue.

It does not.

-Jonathan

 

[Peter Hansen]

I'm working on an application that makes heavy use of Queue objects in
a multithreaded environment.

By "heavy" I mean "millions of calls to put and get, constituting ~20%
of the app's run time." The profiler thinks that a significant amount
of time is spent in this code -- not just a consumer waiting for a
producer, but actual _empty, notify calls, etc.

I wonder if the use case would support hand-crafting an alternative.
Queues appear fairly heavy weight (when you look at the implementation),
and while they are very robust, if your own needs allow putting many
items in at the same time, or getting many items out, for example, then
perhaps you could come up with a much faster alternative based on the
primitives (e.g. Event, Condition, etc) and perhaps some wrapped data
structure other than the list that Queues use.

-Peter

 

[Jason Lai]

As far as Queues go, the adding/popping is apparently done with deque
which are implemented in C. The Queue class pretty much just provides
blocking operations and is otherwise a very thin layer around deque. As
far as primitives go, only threading.Lock is written in C and the
others are pure Python, so they're not that fast, which might be a
reason for Queue's slowness.

As far as writing a custom C module, you could probably leave most of
the work to deque and just implement blocking. If you stick to a simple
lock primative, you can keep it portable and use Python's abstracted
thread interface with an amazing choice of two whole functions: acquire
and release.

 

[Peter Hansen]

As far as Queues go, the adding/popping is apparently done with deque
which are implemented in C. The Queue class pretty much just provides
blocking operations and is otherwise a very thin layer around deque. As
far as primitives go, only threading.Lock is written in C and the
others are pure Python, so they're not that fast, which might be a
reason for Queue's slowness.

BTW, I didn't mean (or expect) that one could improve significantly on
Queue's performance in general just by re-writing it.

I meant that if the OP took into account _his own unique situation_, he
might see opportunities for improvement which Queue couldn't possibly
provide, given that it's a general purpose tool. If his own situation
simply involves a massive number of discrete put/get operations which
cannot be bundled in any fashion, rewriting the entire Queue in C just
to avoid the Python method calls (which have high setup overhead) might
be the only real option.

-Peter