ANN: Kamaelia 0.2.0 released!

[Michael Sparks]

Kamaelia 0.2.0 has been released!

What is it?
===========
Kamaelia is a collection of Axon components designed for network
protocol experimentation in a single threaded, select based environment.
Axon components are python generators are augmented by inbox and outbox
queues (lists) for communication in a communicating sequential processes
(CSP) like fashion.

The architecture is specifically designed to try and simplify the
process of designing and experimenting with new network protocols in
real environments. More background on the motivations behind Kamaelia
can be found here: http://kamaelia.sourceforge.net/Challenges/

The focus of this release adds in support for introspection, pygame based
interfaces, 4 new examples using these, visualisation tools, as well as
syntactic sugar to make building Kamaelia systems simpler. (Specifically
Graphline and pipeline systems. This build upon the existing base allowing
TCP and Multicast based client/server systems.

Other additions and changes include updated examples, variety of bugfixes
in existing components (some pre-emptively discovered by introspection), and
a variety of utility components. It is now also possible to write components
using threading as the concurrency model rather than generators - allowing
integration with thread/blocking only based systems.

The system is known to work under Linux, Mac OS X, Windows and a subset
has been tested on Series 60 mobiles.

General feedback is welcome either directly, mailing lists or via the
project weblog which is here:
* http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi

What's new in version 0.2.0 ?
=============================

Lots! Full release notes can be found here:
* http://kamaelia.sourceforge.net/Kamaelia-0.2.0-ReleaseNotes.html

Editted highlights...

Debian Packages! Many thanks are due to Gintautas Miliauskas, Programmers
of Vilnius, for assistance in building Debian packages. (The current
packages are based on his, and any errors are likely to be mine, not
his) These have been tested successfully on Ubuntu 5.04.

4 new examples have been added showing of various new subsystem:
* Example 5 : This creates a simple streaming system, and looks inside
to see what components are running/active, and passes the resulting
information an Axon Visualiser.

* Example 6 : This is a simple/generic topology visualisation server.
Accepts the following commands over the network
ADD NODE id label auto -
ADD LINK id id
DEL NODE id
DEL ALL
As this stands this is pretty useful, but that's pretty much everything
it does like this.

* Example 7 : This shows how the visualisation subsystem can be extended
to work in different ways. What this does by default when run is
randomly create new nodes and new linkages quite quickly, allowing
you to see how the system works.

* Example 8 : Sample slideshow/presentation tool. Unlike traditional
slideshow/presentation tools, you can modify this to run arbitrary
components. An example of how this can work is provided - allowing
stepping through some graph visualisations along with the presentation.

A Tools directoy has been added with the following tools:
* Axon Shell. (Requires IPython) Implements a simple command line shell
which allows experimentation with Axon systems - the shell runs a
scheduler as a background thread. For a tutorial of use, see:
* http://kamaelia.sourceforge.net/AxonShell.html
* Axon Visualiser. Implements a simple tool for looking inside (quite
literally) running Axon/Kamaelia systems. This allows a very different
style of debugging and can be extremely useful. Tutorial on its way!

Graphlines and Pipelines

These are probably the most useful additions to Kamaelia since 0.1.2.
They are essentially syntactic sugar for building and working with
systems of components, but make building interesting systems rapidly
out of pre-existing components fun and easy. The pipelines follow the
same sort of model as the Unix pipeline. Graphlines are something new,
and like pipelines and all linkages may take any data along them.

Please take a look at the release notes for a graphline example.

A couple of simple pipelines looks like this:

pipeline(
ReadFileAdaptor(file_to_stream, readmode="bitrate", bitrate=400000,
chunkrate=50),
blockise(), # Ensure chunks small enough for multicasting!
Multicast_transceiver("0.0.0.0", 0, "224.168.2.9", 1600),
).activate()
pipeline(
Multicast_transceiver("0.0.0.0", 1600, "224.168.2.9", 0),
detuple(1),
VorbisDecode(),
AOAudioPlaybackAdaptor(),
).run()

A selection of other subsystems have been added - targeted at visualisation
of Kamaelia (and other graph structured) systems using pygame. The layout
mechanism is a simple physics engine.

Key packages of note added: Kamaelia.UI, Kamaelia.UI.Pygame,
Kamaelia.Physics, Kamaelia.Visualisation.Axon,
Kamaelia.Visualisation.PhysicsGraph

Requirements
============
* Python 2.3 or higher recommended, though please do report any bugs
with 2.2.
* Axon (1.1.1 recommended)
* vorbissimple (if you want to use the vorbis decode component/examples)

(Both Axon and vorbissimple are separate parts of the Kamaelia project,
and available at the same download location - see below)

Platforms
=========
Kamaelia has been used successfully under both Linux, Windows and Mac OS
X (panther). A subset of Kamaelia has been successfully tested on Series 60
Nokia mobiles when used with the Axon SERIES 60 branch.

Where can I get it?
===================
Web pages are here:
http://kamaelia.sourceforge.net/Docs/
http://kamaelia.sourceforge.net/ (includes info on mailing lists)

ViewCVS access is available here:
http://cvs.sourceforge.net/viewcvs.py/kamaelia/

Weblog
* http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi

Licensing
=========
Kamaelia is released under the Mozilla tri-license scheme
(MPL1.1/GPL2.0/LGPL2.1). See http://kamaelia.sourceforge.net/Licensing.html

Best Regards,


Michael.
--
(e-mail address removed), http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

 

[phil hunt]

Kamaelia seems it might be an interesting project. However, I don't
think the project is well served by this announcement -- which I
find vague and hard to understand. Which is a shame, because it
means that other people probably don't understand it very well
either, which means less people will use it.

In a spirit of constructive criticism I'd like to give some detailed
comments about this...

Kamaelia is a collection of Axon components designed for network
protocol experimentation in a single threaded, select based environment.
Axon components are python generators are augmented by inbox and outbox
queues (lists) for communication in a communicating sequential processes
(CSP) like fashion.

I really have very little idea what this means.

The architecture is specifically designed to try and simplify the
process of designing and experimenting with new network protocols in
real environments.

The information I can guess out of this is: "Kamaelia is a library
for creating network protocols that sit on top of tcp and/or udp".
Given that it's a BBC project, I imagine we're talking about
protocols for transferring large amount of data, e.g. sound or
motion pictures.

More background on the motivations behind Kamaelia
can be found here: http://kamaelia.sourceforge.net/Challenges/

There's something interesting here:

In Building Public Value, the BBC has committed to the delivery of
BBC content to the British Public over this multitude of systems,
in a way that is enjoyable, accessible and **repurposable** by the
British Public in the way they choose.

(my emphasis)

This is a laudable aim, but if the BBC want to do this, the first
step they could make is to open up their content under a Creative
Commons licence. They could start with their news, current affairs
and documentaries. They could also do this with music downloads: for
example a few weeks ago they made available for free download
all of Beethoven's symphonies, performed by the BBC Orchestra. But the
download only staryed up for a week, and they didn't put the files
under a CC license, so they could be redistributed. Why not? It
would have cost them no more to do so than what they did, and
would've provided better value for downloaders and music lovers.

Of course, the music industry would've complained, but they did
anyway. Frankly, **** them. Their business model is obsolete, and
they should either get a new one or crawl into a corner and die.

BTW, the html is broken in that it's very poorly written, doedsn't
use CSS, and some of the stuff that is intended to display in a
monospaced font doesn't. For example, this HTML code from
<http://kamaelia.sourceforge.net/Docs/Axon.html>:


<div><span style="font-family:Courier 10
Pitch;font-weight:600">def</span><span style="font-family:Courier
10 Pitch"> main(self):</span></div>
<ul type="disc"><div><span style="font-family:Courier 10
Pitch;font-weight:600">yield</span><span
style="font-family:Courier 10 Pitch"> 1<br /></span><span
style="font-family:Courier 10 Pitch;font-weight:600">while</span>
<span style="font-family:Courier 10 Pitch">(self.i):</span></div>

could be better written as:

<pre>
def main(self):
yield 1
while (self.i):
</pre>

Much, much simpler, and actually works.

General feedback is welcome either directly, mailing lists or via the
project weblog which is here:
* http://kamaelia.sourceforge.net/cgi-bin/blog/blog.cgi

I guess the main question that needs to be answered is "Why would I,
as a programmer, use Kamaelia?"

 

[Michael Sparks]

Kamaelia seems it might be an interesting project. However, I don't
think the project is well served by this announcement -- which I
find vague and hard to understand. Which is a shame, because it
means that other people probably don't understand it very well
either, which means less people will use it.

It is a shame, and thanks for mentioning this. Let me have another
go (ripping to shreds welcome

OK, here's a better go. (It's always difficult to think where to pitch
this sort of thing)
---START---
The project aims to make it simple to build networked multimedia
systems (eg audio, video, interactive systems), which are naturally
componentised and naturally concurrent allowing quick and fast reuse in
the same way as Unix pipelines do.

It is designed as a practical toolkit, such that you can build systems
such as:
* Ogg Vorbis streaming server/client systems (via vorbissimple)
* Simple network aware games (via pygame)
* Quickly build TCP based network servers and clients
* Quickly build Multicast based network servers and clients

It runs on Linux, Window, Mac OS X with a subset running on Series 60
phones.

The basic underlying metaphor of a component us like an office worker
with inboxes and outboxes, with deliveries occuring between desks,
offices, and depts. The component can thus do work anyway it likes but
only communicates with these inboxes and outboxes. Like office workers,
components run in parallel, and to achieve this are generally
implemented using python generators, but can also used threads.

The rationale behind the project is to provide a toolkit enabling the
development of new protocols, including streaming, for large scale
media delivery. The license essentially allows use in proprietary
systems without change, but all changes to the system itself must be
shared.
---END---

Is that clearer ?

A short summary of all that could be:

"""Kamaelia is a networking/communications infrastructure for innovative
multimedia systems. Kamaelia uses a component architecture designed to
simplify creation and testing of new protocols and large scale media
delivery systems."""

Hopefully that's clearer than:

[ which you noted "I really have very little idea what this means." ]

The information I can guess out of this is: "Kamaelia is a library
for creating network protocols that sit on top of tcp and/or udp".
Given that it's a BBC project, I imagine we're talking about
protocols for transferring large amount of data, e.g. sound or
motion pictures.

Indeed. However, if you're having to guess I'm not doing the best job
(despite trying!) at explaining this.

There's something interesting here:

In Building Public Value, the BBC has committed to the delivery of
BBC content to the British Public over this multitude of systems,
in a way that is enjoyable, accessible and **repurposable** by the
British Public in the way they choose.

(my emphasis)

This is a laudable aim, but if the BBC want to do this, the first
step they could make is to open up their content under a Creative
Commons licence.

<not such a wise thing for me to reply to, but please read the
disclaimer at the bottom, these are personal opinions only>

I would suggest that you forward this comment directly to the team
involved in working on the Creative Archive. You can do this via this
URL:
* http://creativearchive.bbc.co.uk/

This is unfortunately not as simple as "just putting stuff up", and any
comments you have, I'm sure would be welcome. However, the Creative
Archive hasn't yet been put to a Public Value Test (including impact on
industry), and rights is a much more problematic issue than you might
expect. (eg background music on the radio in eastenders, posters on
people's walls...)

There's also http://www.bbc.co.uk/info/contactus/form.shtml which might
be more appropriate.

On a positive note, the Creative Archive has more organisations involved
than just the BBC, it inlucde the BFI , Channel 4, Open University &
Teachers'TV, so even if these things take longer than some people might
like they do seem to be moving. Public organisations are weighed down
whether the should do something, as well as whether they can.

BBC R&D's role in this is to make these decisions possible.

They could start with their news, current affairs
and documentaries.

BBC Backstage is a start here - at least with some of the online
content. (http://backstage.bbc.co.uk)

But the download only staryed up for a week, and they didn't put the
files under a CC license, so they could be redistributed. Why not? It
would have cost them no more to do so than what they did, and
would've provided better value for downloaders and music lovers.

I can pass on that request if you're interested in an answer - I'm not
personally aware of the rationale behind it only being up for a week.

</not such a wise thing for me to reply to, but please read the
disclaimer at the bottom, these are personal opinions only>

BTW, the html is broken in that it's very poorly written,

It's written using a PyQT based WYSIWYG editor (written on my time,
not the BBC's) that has the limitation that preformatted text gets
squished (since the editor doesn't support the pre tag - or indeed
a lot of other tags). At some point I'll fix the editor to handle
pre-formatted text, but it's not a priority right now. (Should it be?)

The HTML isn't really *that* bad as far as generated code is concerned
IMHO, and is directly what the PyQT QTextEdit widget creates. (It's not
that great either - you may notice it doesn't have proper headings
either)

doesn't use CSS,

This isn't a problem IMHO, YMMV. The site's been tested on a variety of
browsers/platforms and looks pretty much the same on all.

I guess the main question that needs to be answered is "Why would I,
as a programmer, use Kamaelia?"

Probably the most interesting response I've had back from people who've
picked it up is that it's FUN. You can write small sketches (throwaway
code fragments), rewrite their inputs/outputs to take data from inboxes
& outboxes, and then it's usable in a much wider system. It encourages
reuse of code, and assimilating existing systems can be relatively
trivial.

For example, trivial sketches:
* A program to display an image using pygame
* A program that can understand when an area of space has been
clicked. (call it a button)
* A program that can print a list of filenames, pausing waiting for a
user to press return. (call it a chooser)

Trivial changes:
* Take the filename of the image from an inbox
* Rather than print the filenames to send the filename to an outbox.
* Rather than wait for a user to press a key, wait for a message on
an inbox.

A useful composition linking some of these together:

Graphline(
CHOOSER = Chooser(items = files),
IMAGE = Image(size=(800,600), position=(8,48)),
NEXT = Button(caption="Next", msg="NEXT", position=(72,8)),
PREVIOUS = Button(caption="Previous", msg="PREV",position=(8,8)),
linkages = {
("NEXT","outbox") : ("CHOOSER","inbox"),
("PREVIOUS","outbox") : ("CHOOSER","inbox"),
("CHOOSER","outbox") : ("IMAGE","inbox"),
}
).run()

And you have a simple presentation tool !

Another example. The visualisation framework was originally a stand
alone piece of code and was essentially just a piece of eye candy. I
threw in a few yields into certain locations. File reading & static
graph construction was replaced with recieving data from inboxes, and
then it was available for use (and networked after creating trivial
protocol components).

A piece of eye candy is now a useful tool. It's likely to gain the
ability to send to an outbox a message saying "this blob is clicked"
meaning we can have 'WYSIWYG' GUI based construction of pipelines
easily.

It's designed literally as a tool for making it simple to bolt things
together with minimal change. (Which means you choose what you
use, not the framework.)

The system also lends itself to test driven and network independent
development, something not that simple normally with network
protocols.

Why wouldn't you use it? When Twisted is appropriate (Twisted is a more
mature framework).


Michael.
--
(e-mail address removed), http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) contains personal views
which are not the views of the BBC unless specifically stated.
(THIS IS NORMALLY AT THE BOTTON OF MY EMAILS, BUT IT'S REALLY, REALLY,
REALLY THE CASE THIS TIME!!!)

 

[phil hunt]

It is a shame, and thanks for mentioning this. Let me have another
go (ripping to shreds welcome

OK, here's a better go. (It's always difficult to think where to pitch
this sort of thing)
---START---
The project aims to make it simple to build networked multimedia
systems (eg audio, video, interactive systems),

There's plenty of software that facilitates networking, for example
Python already has software for tcp and http clients/servers, and
for xmlrpc remote procedure calls.

So what does Kamaelia do that's extra? I imagine it's to to with
streaming large amounts of data. For example, a streaming video or
audio player. Or VoIP, perhaps.

which are naturally
componentised and naturally concurrent allowing quick and fast reuse in
the same way as Unix pipelines do.

OK, so what do the components in the pipelines do? What sort of data
do they hold? Unix pipelines act on ascii files; I assume you are
do this on audio and visual data. What langauage will the ele,ments
in thne pipelines be written it? I assume some will be in C/C++ for
speed.

It is designed as a practical toolkit, such that you can build systems
such as:

When you say "you" who do you mean? Is the audience programmers or
less technical people? A project that allows non-technical people
to build complex network applications is an ambitious one, but not
impossible (I'd find it very impressive and very exciting,
particularly if it runs on devices such as mobile phones).

* Ogg Vorbis streaming server/client systems (via vorbissimple)
* Simple network aware games (via pygame)
* Quickly build TCP based network servers and clients

What sort of servers and clients? Serving what? Could I use it, for
example, to build a

* Quickly build Multicast based network servers and clients

Serving what? Could I use it, for example, to build an n-player
encrypted (I mean proper encryption here, the sort GCHQ or the NSA
can't break) VoIP server to allow people to do conference calls over
the Internet?

It runs on Linux, Window, Mac OS X with a subset running on Series 60
phones.

The basic underlying metaphor of a component us like an office worker
with inboxes and outboxes, with deliveries occuring between desks,
offices, and depts. The component can thus do work anyway it likes but
only communicates with these inboxes and outboxes. Like office workers,
components run in parallel, and to achieve this are generally
implemented using python generators, but can also used threads.

That metaphor brings up an image (at least to me) that the sorts of
data that can be communicated are things like documents,
spreadsheets, business graphs, memos. Also the metaphor doesn't
indicate any sense of real-time-ness; stuff on an in tray isn't
immediately processed and moved to the out tray.I'm pretty certain
that's not what you have in mind.

May I suggest a different metaphor? Consider a hi-fi system where
you plug cables in to control the dataflow. For example, an
unamplified audio signal goes into an amp, the amp has various dials
and buttons on it, and an amplified signal comes out. Note that the
signal processing is done in real time.

Is that the sort of thing you are working on?

The rationale behind the project is to provide a toolkit enabling the
development of new protocols, including streaming, for large scale
media delivery.

OK, I get the straming part of it. But what asbout non-streaming
stuff? What other protocols are necessary?

The license essentially allows use in proprietary
systems without change, but all changes to the system itself must be
shared.

Fair enough.

---END---

Is that clearer ?

A short summary of all that could be:

"""Kamaelia is a networking/communications infrastructure for innovative
multimedia systems. Kamaelia uses a component architecture designed to
simplify creation and testing of new protocols and large scale media
delivery systems."""

Hopefully that's clearer than:

[ which you noted "I really have very little idea what this means." ]

Sort of. But it's still a bit vague. For example:

"networking/communications infrastructure" can mean a lot of things.
It doesn't even specify whether it is hardware or software.

"innovative". This actually has two meanings. One is "is new /
allows new things to be built". Another is "meaningless marketing
buzzword" (half ). I assume you mean the former, but some people
may be put off by what sounds like buzzwordspeak.

A good short summary would answer these questions:

* What broad category does the system fall under? Software or
Hardware? (the answer here is software). For what sort of
problem-domain? (here the answer is audio and video)

* What sort of software is it? An end product, useful in itself? Or
a library, from which people can build applications?

* Who is it aimed at? Programmers, less-technical users, the genral
public?

* At what stage of completion is it? This is a question sometimes
not answered well by open source projects, which can cause
disappointment if a project is at an earlier stage of development
than some percieve. And also, a project might have immediate goals
that are quite modest, and long-term goals that are more ambitious.

For example, trivial sketches:
* A program to display an image using pygame
* A program that can understand when an area of space has been
clicked. (call it a button)
* A program that can print a list of filenames, pausing waiting for a
user to press return. (call it a chooser)

Trivial changes:
* Take the filename of the image from an inbox
* Rather than print the filenames to send the filename to an outbox.
* Rather than wait for a user to press a key, wait for a message on
an inbox.

A useful composition linking some of these together:

Graphline(
CHOOSER = Chooser(items = files),
IMAGE = Image(size=(800,600), position=(8,48)),
NEXT = Button(caption="Next", msg="NEXT", position=(72,8)),
PREVIOUS = Button(caption="Previous", msg="PREV",position=(8,8)),
linkages = {
("NEXT","outbox") : ("CHOOSER","inbox"),
("PREVIOUS","outbox") : ("CHOOSER","inbox"),
("CHOOSER","outbox") : ("IMAGE","inbox"),
}
).run()

That's certainly less lines of code than it would take in Tkinter.
And easier to follow.

And you have a simple presentation tool !

Now I'm confused. Is Kamaelia a GUI builder?

Another example. The visualisation framework was originally a stand
alone piece of code and was essentially just a piece of eye candy. I
threw in a few yields into certain locations. File reading & static
graph construction was replaced with recieving data from inboxes, and
then it was available for use (and networked after creating trivial
protocol components).

Ah, so now I'm guessing it allows me to write networked PyGame
applications, where with just a few lines of code I can have
networked streaming video on my pyGame screen... am I right?

A piece of eye candy is now a useful tool. It's likely to gain the
ability to send to an outbox a message saying "this blob is clicked"
meaning we can have 'WYSIWYG' GUI based construction of pipelines
easily.

That sounds useful.

Why wouldn't you use it? When Twisted is appropriate (Twisted is a more
mature framework).

The problem with Twisted, IME, is I don't understand the
documentation.

 

[Sergei Organov]

Unix pipelines act on ascii files;

No, they don't.

 

[phil hunt]

(e-mail address removed) (phil hunt) writes:
[...]

Unix pipelines act on ascii files;

No, they don't.

Nitpicker.

I would have thought it was perfectly obvious, in context, what I
meant.

 

[Michael Sparks]

I've reordered the q's slightly to avoid repetition... Also by answering
this question first, it may put the rest of the answer into context
better.

At what stage of completion is it?

This is something we deliberately try to reflect in the version number.
Yes, you can build network servers and new protocols relatively simply
at the moment. Yes, you can integrate with pygame in a limited useful
fashion at present. Yes we have audio playback.

However we don't have yet... (some examples)
* Decent GUI integration yet.
* /Full/ pygame integration.
* Nice integration with pymedia
* Direct support for Dirac.
Which aside from other things means you can't build (say) a video
& SMIL playback system trivially, yet.

As a result that's why the version number is 0.2 - whilst you /can/ do a
lot, there's a lot more to do. Clearly that also naturally implies that
we don't expect any end user to be looking at the site. (The low
version number will normally scare them away)

There's plenty of software that facilitates networking, for example
Python already has software for tcp and http clients/servers, and
for xmlrpc remote procedure calls.

There is indeed.

So what does Kamaelia do that's extra? I imagine it's to to with
streaming large amounts of data. For example, a streaming video or
audio player. Or VoIP, perhaps.

It's designed to make bolting things together to make these sorts of
system simpler and easier. At the same time it's designed to encourage
writing code in a way that makes it simpler. The natural side effect of
this is the system might make it easier to take advantage of multiple
CPU systems as they come online, since it makes a system naturally
concurrent. As the original announcement said "Kamaelia is designed as
a testbed". And by testbed I mean testbed as it testing out new ideas,
see if they work and see if they pan out. (Not as in a testing suite)

Probably the best way of describing the difference is this... After my
talk about Kamaelia at Europython, I had an long chat with Tommi
Virtinan about integration between Kamaelia and Twisted. I haven't had
a chance to follow up with him yet regarding how this would work, though
I have set a line in the sand aiming to have easy integration between
Kamaelia and Twisted before Kamaelia hits version 1.0.0. The impression
I got from Tommi was that he was most interested in the communications
aspect - the fact we can bolt together systems in a manner directly
akin to Unix pipelines, though I suspect he found the graphines aspect
more interesting.

Or as someone asking a similar question at Open Tech exclaimed after
I finally managed to explain it better to them "Ooooh - you're trying
to make concurrency EASY!".

OK, so what do the components in the pipelines do? What sort of data
do they hold? Unix pipelines act on ascii files; I assume you are
do this on audio and visual data. What langauage will the ele,ments
in thne pipelines be written it? I assume some will be in C/C++ for
speed.

Components are object instances of python classes. The data passed
between components are object instances. Clearly these python classes
can be written in python, C, C++, pyrex etc. Currently all of Kamaelia's
components are python based. Some existing components make calls
into some related C libraries via python bindings.. An example of
writing a component using Pyrex can be found here:
* http://kamaelia.sourceforge.net/PyrexComponents.html

When you say "you" who do you mean?

Generally I expect the readership of c.l.p/[email protected] to be
programmers. Python is generally easy to pick up and having asked
someone who's not done much programming beforehand (beyond
a small amount of VB and Access), and is pre-university to use the
system to build a simple streaming system prototyping visualising PVR
content on a mobile (and watching them succeed), they seem
relatively reasonable examples.

At some point, the ability to allow non-programmers to bolt together
Kamaelia systems would be desirable, but a first step is making it
simpler for programmers to bolt together systems. We currently have an
interactive visualisation tool(*), and the logical extension of that is
a tool that allows systems to be bolted together without knowing any
code.
(*) http://kamaelia.sourceforge.net/AxonVisualiser.html

It'd be an interesting side project for someone to take forward, and
might be low hanging fruit in terms of projects. (Especially if viewed
initially as a tool for assisting development, rather than replacing
development)

Is the audience programmers or
less technical people? A project that allows non-technical people
to build complex network applications is an ambitious one, but not
impossible (I'd find it very impressive and very exciting,
particularly if it runs on devices such as mobile phones).

It's a little ambitious at this stage, yes.

What sort of servers and clients?

Whatever you feel like. If you want a server to split and serve audio,
you could do that. if you want a server to spit out fortune cookies,
you can do that. (Useful only really as an alternative to a chargen
test protocol IMO)

Serving what? Could I use it, for example, to build an n-player
encrypted VoIP server to allow people to do conference calls over
the Internet?

You could do that probably. (Though we don't have a component
for audio capture (though a read file adaptor reading from /dev/audio
might work depending on your platform I suppose) and audio
encoding at the moment, so those would probably be the core
components to integrate. If you want to use multicast over the wide
area internet you'd also have to convince all the people using the
system to use ISPs that support multicast......)

(I mean proper encryption here, the sort GCHQ or the NSA can't break)

I'd be impressed if that could be written, using anything really. (Can't
implies never)

That metaphor brings up an image (at least to me) that the sorts of
data that can be communicated are things like documents,
spreadsheets, business graphs, memos.

They could indeed. The underlying framework doesn't differentiate
between data nor have any realtime aspect embedded in the system
at present. Just because we're focussing on systems that have a realtime
element and are multimedia based, this does not mean the system is
limited to that.

May I suggest a different metaphor? [hifi]

I'll think about it. It's closer to the model that pysonic seems
to take, and it implies a functional transform - ie /just/ dataflow
- rather than connected processing units that /might/ take time
to do processing.

OK, I get the straming part of it. But what asbout non-streaming
stuff? What other protocols are necessary?

One example is peer to peer mesh setup. People normally
think of P2P as a distribution mechanism. However, the underlying
approach also very good at setting up communications meshes.
This could be of use in many areas, such as GRID based systems
for distributed rendering, application layer multicast, and network
multicast island joining.

Due to the illegal /uses/ of P2P, much work in this area is difficult to
reuse due to defensive coding. Decoupling development of protocols
from use of those protocols is generally a wide idea IMO.
[ new descscription followed by useful comments, we'll take
them on board, I think I've answered most points inline ]

"innovative". This actually has two meanings. One is "is new /
allows new things to be built".

Bingo. It could be argued the other is misuse as buzzword.

That's certainly less lines of code than it would take in Tkinter.
And easier to follow.

We're looking at tkinter as well. (Some tests at tkinter integration
are in CVS)

Now I'm confused. Is Kamaelia a GUI builder?

Multimedia systems have many aspects. They include taking in
audio/video and spitting out audio/video/pictures/text/... Take
a look a SMIL if you're curious - it's a system that Kamaelia would be
incomplete if it made decoding/display/interpretation of SMIL
difficult.

We also have to be able to demonstrate system to other people
inside the BBC in a way non-technical people understand. That means
showing structures in a friendly dynamic way, showing pictures,
playing sounds (hence visualisation - looking inside running systems).

That means we need ways of integrating with systems like pygame &
other toolkits. If however I'm talking outside the BBC I'll try to give
examples which people might find interesting - such as building a
presentation tool. The blocks are very much like Lego & K'Nex and
adding in a new block enables all sorts of new applications.

For example, we could take the text ticker, combine that with a text
feed and have a personal autocue/teleprompter. Alternatively someone
could use it to have subtitles (say) at the opera displayed on a Nokia
770 (maemo) based device.

Ah, so now I'm guessing it allows me to write networked PyGame
applications, where with just a few lines of code I can have
networked streaming video on my pyGame screen... am I right?

As yet, no. When we hit version 1.0, yes. Why? Because we haven't
built components for video decode & display (and that's the only
reason). That's why I don't mention video normally - I prefer to focus
on where we are when announcing things, rather than where we're
going (leaving that to more appropriate forums).

The problem with Twisted, IME, is I don't understand the

documentation.

I can't help there, sorry. There *is* a twisted book coming out shortly,
which might be worth looking into (or getting from a library), if
you're interested...

Finally, please note: Kamaelia is part of an R&D project - not all of
the results will pan out. Some might, some might not. Unless you aim
for the Stars though you won't get to Mars.

Best Regards,


Michael.
--
(e-mail address removed), http://kamaelia.sourceforge.net/
British Broadcasting Corporation, Research and Development
Kingswood Warren, Surrey KT20 6NP

This message (and any attachments) may contain personal views
which are not the views of the BBC unless specifically stated.

 

[phil hunt]

I've reordered the q's slightly to avoid repetition... Also by answering
this question first, it may put the rest of the answer into context
better.



This is something we deliberately try to reflect in the version number.
Yes, you can build network servers and new protocols relatively simply
at the moment. Yes, you can integrate with pygame in a limited useful
fashion at present. Yes we have audio playback.

However we don't have yet... (some examples)
* Decent GUI integration yet.
* /Full/ pygame integration.
* Nice integration with pymedia
* Direct support for Dirac.
Which aside from other things means you can't build (say) a video
& SMIL playback system trivially, yet.

Isn't SMIL something that's goinhg to go into web browsers? In which
case, you'd presumably not want to build one yourself, I imagine?

There is indeed.


It's designed to make bolting things together to make these sorts of
system simpler and easier.

What you say "bolting things together" do you mean writing Python
code? Or will there be some other way?

What I have in mind is something like a project I worked on some
time ago, where a program could be written by drawing boxes on a GUI
application, and drawing lines (representing data flow) to connect
the boxes. So one half of a VoIP application might look like:

+----------+ +------------+ +-----------+
| listen | | convert to | | volume |
| on udp |====>| audio |====>| control + |
| port 600 | | stream | | output |
+----------+ +------------+ +-----------+

With something like this novel multimedia apps could be prototyped
quickly (though making them into useful apps would take extra work
-- in the case of a VoIP application you'd want a phonebook, for
example).

At the same time it's designed to encourage
writing code in a way that makes it simpler.

Examples would be useful here. Both of what it can do now, and what
it will be able to do eventually.

The natural side effect of
this is the system might make it easier to take advantage of multiple
CPU systems as they come online, since it makes a system naturally
concurrent. As the original announcement said "Kamaelia is designed as
a testbed". And by testbed I mean testbed as it testing out new ideas,
see if they work and see if they pan out. (Not as in a testing suite)

So what it will eventually do is not cast in stone?

Probably the best way of describing the difference is this... After my
talk about Kamaelia at Europython, I had an long chat with Tommi
Virtinan about integration between Kamaelia and Twisted. I haven't had
a chance to follow up with him yet regarding how this would work, though
I have set a line in the sand aiming to have easy integration between
Kamaelia and Twisted before Kamaelia hits version 1.0.0. The impression
I got from Tommi was that he was most interested in the communications
aspect - the fact we can bolt together systems in a manner directly
akin to Unix pipelines, though I suspect he found the graphines aspect
more interesting.

Or as someone asking a similar question at Open Tech exclaimed after
I finally managed to explain it better to them "Ooooh - you're trying
to make concurrency EASY!".


Components are object instances of python classes. The data passed
between components are object instances.

What sort of objects? For example, if an application is audio
straming, what sort of objects will be passed? Will there be onre
object passed for each 10 ms or so of sound, or will there be a
continuously-set-up object that passes the data throught it a byte
at a time?

And if it's written in Python, will it run fast enough?

Clearly these python classes
can be written in python, C, C++, pyrex etc. Currently all of Kamaelia's
components are python based. Some existing components make calls
into some related C libraries via python bindings.. An example of
writing a component using Pyrex can be found here:
* http://kamaelia.sourceforge.net/PyrexComponents.html

My thought on this example:

cdef class test(Axon.Component.component):
def mainBody(self):
if self.dataReady("inbox"):
data = self.recv("inbox")
self.send(data, "outbox")
return 1

My first impressions are that it seems obvious to me what this does:
it merely passes its input throught to its output (no doubt there
are places where such a null component is useful). Hopefully all
code written with Kamaelia will be so clear.

Now I have some questions.

1. mainBody() is evidently being called at intervals. At what
frequency are these intervals? For example, if the data stram is
video data, will mainBody() get called once for each frame?

2. what data type is (data) in the code? Or can it be any type?

 

[phil hunt]

It's a little ambitious at this stage, yes.

But it couldbe there eventually?

Whatever you feel like. If you want a server to split and serve audio,
you could do that.

This is streaming audio, right? For non-streaming I can just use an
ftp or http server.

You could do that probably. (Though we don't have a component
for audio capture (though a read file adaptor reading from /dev/audio
might work depending on your platform I suppose) and audio
encoding at the moment, so those would probably be the core
components to integrate.

That's a slightly worrying answer for me, worrying because it seems
I've misunderstood the nature of the project. I assumed that
components for audio capture, and related activities, would be at
the heart of the project.

If you want to use multicast over the wide
area internet you'd also have to convince all the people using the
system to use ISPs that support multicast......)

(or just sent the signal out multiple times)

I'd be impressed if that could be written, using anything really. (Can't
implies never)

What -- good encryption? That's pretty much a well-known technique
these days (unless the NSA has some *very* advanced hardware in
their basement, which I strongly suspect they don't).

They could indeed. The underlying framework doesn't differentiate
between data nor have any realtime aspect embedded in the system
at present. Just because we're focussing on systems that have a realtime
element and are multimedia based, this does not mean the system is
limited to that.

Again, this makes me think I've misunderstood the project.

One example is peer to peer mesh setup. People normally
think of P2P as a distribution mechanism. However, the underlying
approach also very good at setting up communications meshes.

When you say a mesh, what do you mean?

This could be of use in many areas, such as GRID based systems
for distributed rendering, application layer multicast, and network
multicast island joining.

Unpack, please.

Due to the illegal /uses/ of P2P, much work in this area is difficult to
reuse due to defensive coding.

Oh. Could you give an example?

We also have to be able to demonstrate system to other people
inside the BBC in a way non-technical people understand. That means
showing structures in a friendly dynamic way, showing pictures,
playing sounds (hence visualisation - looking inside running systems).

Visualisation, if done properly, ought to be useful to technical
people too.

That means we need ways of integrating with systems like pygame &
other toolkits. If however I'm talking outside the BBC I'll try to give
examples which people might find interesting - such as building a
presentation tool. The blocks are very much like Lego & K'Nex and
adding in a new block enables all sorts of new applications.

That's kind of the impression that I've got.

For example, we could take the text ticker, combine that with a text
feed and have a personal autocue/teleprompter. Alternatively someone
could use it to have subtitles (say) at the opera displayed on a Nokia
770 (maemo) based device.

That would be useful.

Or you could have subtitles in different languages, and the user
gets to choose which one to display...

 

[Michael Sparks]

But it couldbe there eventually?

Could? Yes. Will? Can't say. I can agree it would be nice, and given
time/resources (or someone sufficiently interested externally) then it
may happen. (IMO there's no real reason that it couldn't happen aside
from time/effort/resources)

This is streaming audio, right? For non-streaming I can just use an
ftp or http server.

There's more to network servers and clients than just audio & video, or
unidirectional download.

For example the visualisation/introspection tool is an example of a
client server system. The server is the visualisation tool. It listens on a
specified port waiting for a connection. The client connects and sends
data to it about the internal structure, and the server displays this.

That's a slightly worrying answer for me, worrying because it seems
I've misunderstood the nature of the project. I assumed that
components for audio capture, and related activities, would be at
the heart of the project.

*Our* main interest, at the moment, is in /delivery/ of content.

Dealing with capture would be largely re-inventing wheels before we know
whether the framework is a suitable framework. We are looking at making it
possible to use pymedia for dealing with capture/encoding/decoding.

There's a number of things along the way we need to deal with this, but
we're not starting from the perspective of capture.

(After all for capture we can generally look at using dedicated encoder
hardware that will often spit out it's encoded information in the form of a
network connection. As a result capture and encoding hasn't been a priority
as yet. Sometimes looking into a project from the outside I can appreciate
that certain decisions might look strange, but consider that you don't need
to worry about capture in order

What -- good encryption? That's pretty much a well-known technique
these days (unless the NSA has some *very* advanced hardware in
their basement, which I strongly suspect they don't).

You said *can't*. That says to me cannot ever be broken. If you have a large
number of listeners, as your statement implied, that implies decryptable by
many listeners - you then just need one compromised listener (essentially
you're asking for the equivalent of implementing a DRM system that the NSA
couldn't break...).

If you can provide me with a library that you can guarantee that it will
satisfy the following properties:

encoded = F(data)

and a piece of client code that can do this:
decoded = G(encoded)

Then yes, that can be wrapped. That's trivial in fact:
---(start)---
from magic import unbreakable_encryption

class encoder(component):
def __init__(self, **args):
self.encoder = unbreakable_encryption.encoder(**args)
def main(self):
while 1:
if self.dataReady("inbox"):
data = self.recv("inbox")
encoded = self.encoder.encode(data)
self.send(encoded, "outbox")
yield 1

class decoder(component):
def __init__(self, **args):
self.decoder = unbreakable_encryption.decoder(**args)
def main(self):
while 1:
if self.dataReady("inbox"):
data = self.recv("inbox")
decoded = self.decoder.decode(data)
self.send(decoded, "outbox")
yield 1
---(end)---

If you believe you can implement F&G for general use such that F&G
can //never// be decrypted by anyone other than authorised recipients, I
suggest you cease this conversation - you have some highly marketable
technology.

Again, this makes me think I've misunderstood the project.

Realtime systems are a subset of all systems that are interesting in terms
of network delivery & multimedia systems. Realtime scheduling is a well
known area and if/when this becomes an issue, we'll look at adding it into
the mix. The real problem is dealing with concurrency and making it simple
to work with. Making realtime concurrent systems easy to work with strikes
me as running before you can walk.

(Network systems are naturally concurrent, so if you're aiming to make
network systems easy to build you're really talking about making concurrent
systems easy to build.)

When you say a mesh, what do you mean?

I mean a mesh. (see below)

Unpack, please.

They're all big systems, all of which utilise networks of collaborating
systems for different purposes.

Grid starting point:
* http://en.wikipedia.org/wiki/Grid_computing

*short* introduction to application level multicast (has many names,
including overlay multicast):
* http://www.mnlab.cs.depaul.edu/seminar/fall2002/Overcast.pdf
* Also puts the term "mesh" in context.

Multicast island joining is a special case and is exactly what it says -
joining multicast islands together.

Oh. Could you give an example?

How many RFCs have you seen documenting the protocols used by (say)
Napster, Bit Torrent, Gnutella, Limewire, Freenet? The legitimate uses of
Bit Torrent for example tend to get ignored by certain large companies
when trying to shut down systems.

Visualisation, if done properly, ought to be useful to technical
people too.

It is (as mention on the page describing the visualisation tool).

Regards,


Michael.

 

[Michael Sparks]

....

Isn't SMIL something that's goinhg to go into web browsers? In which
case, you'd presumably not want to build one yourself, I imagine?

I was using SMIL as an example to be illustrative of a class of networked
applications that require visual support. A more direct example
would be MHEG (not mpeg) decoding and display systems, which
have similarities to SMIL systems in terms of capabilities required.

What you say "bolting things together" do you mean writing Python
code?
Yes.

Or will there be some other way?

A clicky pointy approach would be nice. (There's two sorts of interface in
the world IMO - "tappity tappity" and "clicky pointy" - if being equally
disdainful Unix shell systems are very "tappity tappity", and whilst I
can do "tappity tappity", and have been doing "tappity tappity" for my
entiring working life, I do prefer "clicky pointy" systems.)

Though building systems using AR Toolkit type systems would be the ideal.
( http://www.hitl.washington.edu/people/poup/research/ar.htm#artoolkit )
That's probably a couple of years off though (at least).

What I have in mind is something like a project I worked on some
time ago, where a program could be written by drawing boxes on a GUI
application, and drawing lines (representing data flow) to connect
the boxes. So one half of a VoIP application might look like:

+----------+ +------------+ +-----------+
| listen | | convert to | | volume |
| on udp |====>| audio |====>| control + |
| port 600 | | stream | | output |
+----------+ +------------+ +-----------+

From example 4:

pipeline(
Multicast_transceiver("0.0.0.0", 1600, "224.168.2.9", 0),
detuple(1),
SRM_Receiver(),
detuple(1),
VorbisDecode(),
AOAudioPlaybackAdaptor(),
).run()

Join multicast group "224.168.2.9", listen for packets on port 1600, throw
away information relating to who sent it, pass it through a simple reliable
multicast reordering subsystem, throw away the annotation, decode the
vorbis, and playback.

The visual representation of that isn't dissimilar to your diagram above.

With something like this novel multimedia apps could be prototyped
quickly (though making them into useful apps would take extra work
-- in the case of a VoIP application you'd want a phonebook, for
example).
Agreed.


Examples would be useful here. Both of what it can do now, and what
it will be able to do eventually.

Examples of what can be done now are here:
* http://kamaelia.sourceforge.net/KamaeliaStatus.html

I'll think about the latter part of that. (I personally loath technologies
that promise much and deliver little. Hence why I like to focus on what
we /can/ do)

So what it will eventually do is not cast in stone?

The high level goals are to solve the problems in the Challenges document.
We're not particularly wedded to any one particular approach, and Kamaelia
can/should be moulded to solve those problems, not the other way round.

By definition along the way a variety of components will be generated, which
can obviously be put together in an arbitrary fashion, assuming type
adaptors as necessary, to form new interesting systems.

This ability to explore new/interesting systems quickly is something
generally of value to an R&D dept, so there's some people at BBC R&D
interested in using Kamaelia for all sorts of interesting projects. (which I
can't discuss here)

What sort of objects?

Python objects. Numbers, strings, chunks of audio in a string, chunks of
audio in a binary object wrapped by a python class, etc. Precisely as I
said.

For example, if an application is audio
straming, what sort of objects will be passed?

That depends on the level to which you dive inside the audio being
streamed. If you're trying to be a content agnostic streamer then you
don't care what's inside the byte stream (from, say, a file) just what the
bit rate is you're supposed to send at. (for that you can just chop up the
file and send it chunks at a time)

If however you're dealing with variable bit rate data you may wish to look
inside the wrapper to figure out the where data blocks are for a given time
period in order to send those. This (for example in the case of sending MPEG
over RTP) can mean that you need to parse the data stream in order to send
the data. At that point it makes sense to send over a structured object
representing each chunk/block/frame (depending on codec/transport).

Will there be onre
object passed for each 10 ms or so of sound, or will there be a
continuously-set-up object that passes the data throught it a byte
at a time?

Small messages turn out to be inefficient for what it's worth. (It's more
efficient to send bundles of small message or larger messages)

And if it's written in Python, will it run fast enough?

I've seen overlay mesh networks implemented in TCL shipping 60Mbit/s
(sustained) in the past, which if we're implementing an overlay joining
multicast islands together would mean multicasting the equivalent of ~200
channels through each node. Scaling is not always about throughput of
course - Bit Torrent's initial implementation was also in python, and scales
very well.

If it doesn't scale sufficiently we can always optimise specific subsystems
when we need to rather than prematurely optimise. Also we have done a
naive (deliberately naive) translation of the ideas into C++, including
generators, so if things don't pan out we have a natural migration path else
where. (After knowing whether the overall approach is workable or not)

cdef class test(Axon.Component.component):
def mainBody(self):
if self.dataReady("inbox"):
data = self.recv("inbox")
self.send(data, "outbox")
return 1

My first impressions are that it seems obvious to me what this does:

A line of code is more often read than written, so if it's obvious, that's
good.

it merely passes its input throught to its output
Correct.

(no doubt there are places where such a null component is useful).

An echo protocol is one example - a server you connect to that simply echoes
back to you whatever you send it. (an echo server can be useful for all
sorts of network testing in practice)

Hopefully all code written with Kamaelia will be so clear.

See here for a longer example - which should show that generally speaking
things do tend to remain clear: http://tinyurl.com/dp8n7 (also shows how
you can incrementally develop new components).

Now I have some questions.

1. mainBody() is evidently being called at intervals. At what
frequency are these intervals? For example, if the data stram is
video data, will mainBody() get called once for each frame?

The scheduler is currently a dumb scheduler with essentially the following
logic:
for thread in microthreads:
thread.next()

As a result the time between calls is not guaranteed (nor can it be with
this sort of scheduler). The system does however allow for multiple
schedulers, so you could potentially have a realtime scheduler if it
was found to be necessary, at which point information about how a
component wished to be scheduled would become important.

2. what data type is (data) in the code? Or can it be any type?

Any data type python handles. (as indicated above)

In case you're wondering "why doesn't it do everything yet", I'll
just simply reiterate that we're at version 0.2, and we're getting
to the crawling/walking stage first before the walking/talking/running
stage

Best Regards,


Michael.

 

[Wolfgang Keller]

The project aims to make it simple to build networked multimedia

systems (eg audio, video, interactive systems),

Why limit the scope of application artificially?

Axon/Kamaelia imho is perfectly applicable to any kind of application
that does some kind of data processing.

Think workflow applications, business applications, application
integration...

J. Paul Morrison has used the concept he named "Flow-Based-Programming"
for financial applications since the 60's, any control systems engineer
knows function-block-diagrams etc. and so on...

Sincerely,

Wolfgang Keller

 

[Matt Hammond]

Why limit the scope of application artificially?

Axon/Kamaelia imho is perfectly applicable to any kind of application
that does some kind of data processing.

You're absolutely right - Axon/Kamaelia hopefully will not just be limited
to those areas. It just so happens that those kinds of systems are the
underlying reasons why we started building Axon/Kamaelia. That's therefore
where we're concentrating our resources at the moment.

Think workflow applications, business applications, application
integration...

J. Paul Morrison has used the concept he named "Flow-Based-Programming"
for financial applications since the 60's, any control systems engineer
knows function-block-diagrams etc. and so on...

We have ... alot! I really do hope that this system is used for more. I'd
be very interested to know if anyone has had a go at building any
components - for whatever purpose. The potential for component re-use
might mean less work for us!


regards


Matt Hammond