Python + Shoutpy + Twisted Locks

[exhuma.twn]

Unfortunately I don't have the code at hand on this box, but maybe
someone can give me a nudge in the right direction.

Some background: Last year I began to write a jukebox system that
provided a Telnet-like interface. I wrote this using "socket". Later
along the path I discovered Twisted, and due to various other
problems, a re-write made sense. So I decided to use Twisted. The only
thing I adapted was the class providing the telnet interface. In the
background I use a few other threads. One for metadata-scanning, one
for assuring continuous playback (periodic checks the player status
and appends songs if necessary) and in the case the shoutcast backend
is activated, there is also a built-in player thread.

I remember having it running already, but I don't remember if it was
before I started to use Twisted or after.

Well, I beleive the problem lies withing the last named thread. This
one uses shoutpy to stream data to the shoutcast server. What I found
is that "libshout" is blocking, which should be fine as the whole
thing runs in it's separate thread. But the application hangs
nevertheless while streaming. This effectively blocks out the other
thread that checks the player status, which then fails to append new
songs to the queue. So only one song is played when streaming.

I saw that there is another package called "python-shout". But neither
that nor "shoutpy" have any comprehensive documentation which might
tell me anything about the libshout blocking problem. The
documentation of libshout revealed a "non-blocking" mode, but even
though I made the necessary adjustments to the code, I still had the
same problem.

The other threads in my application run fine and don't block the rest
of the app. So I guess, that the main problem is that blocking occurs
"outside" the python world and "inside" the libshout world. Maybe the
combination of all this with Twisted is an unlucky marriage but I
would be disappointed to be told: "Nope, won't be possible"

Maybe Twisted's deferreds to the rescue? I don't know... I'm still
quite green with Twisted

 

[Gabriel Genellina]

[...] What I found
is that "libshout" is blocking, which should be fine as the whole
thing runs in it's separate thread. But the application hangs
nevertheless while streaming. This effectively blocks out the other
thread that checks the player status, which then fails to append new
songs to the queue. So only one song is played when streaming.

The other threads in my application run fine and don't block the rest
of the app. So I guess, that the main problem is that blocking occurs
"outside" the python world and "inside" the libshout world.

Only one thread at a time may be executing Python code; the Global
Interpreter Lock (GIL) ensures the mutual exclusion. Extension modules
(written in C) may use the macros
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release/acquire the GIL
before/after an external blocking call.
I don't know libshout, or how you are doing the binding python-libshout,
but if your analysis is correct it means that the code is not releasing
the GIL at the appropiate points.

 

[Michel Albert]

[...] What I found
is that "libshout" is blocking, which should be fine as the whole
thing runs in it's separate thread. But the application hangs
nevertheless while streaming. This effectively blocks out the other
thread that checks the player status, which then fails to append new
songs to the queue. So only one song is played when streaming.

The other threads in my application run fine and don't block the rest
of the app. So I guess, that the main problem is that blocking occurs
"outside" the python world and "inside" the libshout world.

Only one thread at a time may be executing Python code; the Global
Interpreter Lock (GIL) ensures the mutual exclusion. Extension modules
(written in C) may use the macros
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release/acquire the GIL
before/after an external blocking call.
I don't know libshout, or how you are doing the binding python-libshout,
but if your analysis is correct it means that the code is not releasing
the GIL at the appropiate points.

Hmmm... ok. I suppose rewriting the whole thing using twisted's
deferreds could then solve the problem. Which are basically nothing
more than callbacks with a weird name Unfortunately this means that
I have to re-think a lot. But in the end I suppose it will pay off.

Thanks for taking the time and reading my little essay Gabriel

 

[Chris Mellon]

[...] What I found
is that "libshout" is blocking, which should be fine as the whole
thing runs in it's separate thread. But the application hangs
nevertheless while streaming. This effectively blocks out the other
thread that checks the player status, which then fails to append new
songs to the queue. So only one song is played when streaming.

The other threads in my application run fine and don't block the rest
of the app. So I guess, that the main problem is that blocking occurs
"outside" the python world and "inside" the libshout world.

Only one thread at a time may be executing Python code; the Global
Interpreter Lock (GIL) ensures the mutual exclusion. Extension modules
(written in C) may use the macros
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release/acquire the GIL
before/after an external blocking call.
I don't know libshout, or how you are doing the binding python-libshout,
but if your analysis is correct it means that the code is not releasing
the GIL at the appropiate points.

Hmmm... ok. I suppose rewriting the whole thing using twisted's
deferreds could then solve the problem. Which are basically nothing
more than callbacks with a weird name Unfortunately this means that
I have to re-think a lot. But in the end I suppose it will pay off.

Thanks for taking the time and reading my little essay Gabriel

Using Twisted won't help if the libshout calls are really blocking the
main thread. You need to fix the libshout wrapper to release the GIL
when it makes blocking C calls.

I also believe that Twisted has a native (perhaps only
semi-functional?) shoutcast protocol implementation, so a pure-python
implementation might be possible as well.

 

[John Nagle]

En Fri, 05 Oct 2007 04:55:55 -0300, exhuma.twn <[email protected]> escribi?:

[...] What I found
is that "libshout" is blocking, which should be fine as the whole
thing runs in it's separate thread. But the application hangs
nevertheless while streaming. This effectively blocks out the other
thread that checks the player status, which then fails to append new
songs to the queue. So only one song is played when streaming.
The other threads in my application run fine and don't block the rest
of the app. So I guess, that the main problem is that blocking occurs
"outside" the python world and "inside" the libshout world.
Only one thread at a time may be executing Python code; the Global
Interpreter Lock (GIL) ensures the mutual exclusion. Extension modules
(written in C) may use the macros
Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS to release/acquire the GIL
before/after an external blocking call.
I don't know libshout, or how you are doing the binding python-libshout,
but if your analysis is correct it means that the code is not releasing
the GIL at the appropiate points.

Hmmm... ok. I suppose rewriting the whole thing using twisted's
deferreds could then solve the problem. Which are basically nothing
more than callbacks with a weird name Unfortunately this means that
I have to re-think a lot. But in the end I suppose it will pay off.

Thanks for taking the time and reading my little essay Gabriel

Using Twisted won't help if the libshout calls are really blocking the
main thread.

Right.

The whole point of Twisted is that you can't do anything that takes
very long in a Twisted thread, or the whole server will stall. If you
try to do something ongoing, like streaming, in a Twisted server, you
have to have long-running threads of your own and a scheduling system
to coordinate them with the Twisted threads.

libshout's "nonblocking" mode isn't very nonblocking. See

https://trac.xiph.org/browser/icecast/trunk/libshout/examples/nonblocking.c

There's a thread in a loop during playout. All "nonblocking" seems to do is
defer the waiting from shout_write() to shout_sync().

It looks like you'll need a separate thread for each stream being played
out, on top of the Twisted machinery. Or a completely event-driven version
of "libshout" designed for Twisted.

John Nagle