Request for comments on python distributed technologies

[Carl J. Van Arsdall]

Hey everyone, another question for the list. In particular i'm looking
for comments on some of the distributed technologies supported in
python. Specifically, I'm looking at XML-RPC, RPyC, CORBA, and Twisted.

Before you offer any comments let me talk about what i'm doing a little
bit. Right now I'm creating an execution framework. The framework aims
to allow users to create a modular system by which engineers can
mitigate some of the issues seen in large complex systems. By using
modules the system hopes to allow the system to change but allow
engineers to re-use as much code as possible. By using the modules in a
framework we hope to encapsulate the entire system to that it is
controllable in the sense that we can generate and catch errors anywhere
in the system and exit gracefully, things like that. There's a lot more
to it but the motivation spawn from our system where the system kept
needing to adapt to support changes in our company. These changes over
time have resulted in a large number of "hacks". The hope of the
modules is to make the code a lot cleaner and organized so that the
system can operate in whatever way it needs to without having so many
hacks. This system I use also operates in a primitive distributed
environment. Currently, these are ssh calls to other nodes that then
execute other python scripts and shell commands. This is the next piece
I'm interested in moving away from.

With all that being said, I've come across those four technologies so
far and I've been looking into them. I was wondering if anyone had any
comments, pros/cons, or suggestions of other distributed technologies to
take a look at. I intend to integrate the distributed technology
directly into the framework making it easy for developers to distribute
tasks. Anyhow, any comments would be greatly appreciated! I also hope
to release this framework as an open-source project when its finished,
so again I community input is important.

Thanks!

..c


--

Carl J. Van Arsdall
(e-mail address removed)
Build and Release
MontaVista Software

 

[Paul McGuire]

Hey everyone, another question for the list. In particular i'm looking
for comments on some of the distributed technologies supported in
python. Specifically, I'm looking at XML-RPC, RPyC, CORBA, and Twisted.

What you describe sounds very much like pyro, which should probably be added
to your list.

BTW, distributed frameworks are not inherently more robust or more easily
maintained than monolithic systems. Unless you establish dynamic
replication and/or relocation of services, you can easily end up with one or
more single points of failure. Services should be self-registering in order
to make finding them an automatic process, without having to maintain global
configuration/locator files or services (which themselves can become
bottlenecks), or distributed lookup files that must be propagated when
changes occur (with appropriate change collision management). You also want
to incorporate some form of isolation in your framework, to try to avoid
permitting ill-behaved clients from crashing services. A global logging
service for all services is handy - its a bear traipsing through and
interleaving among log files from each separate service node - but this too
can be a drag on performance and robustness. And also, don't forget admin
and monitoring utilities - some op center somewhere is going to have to
monitor this whole mess^H^H^H^Hsystem, and would like an early warning
before the one critical bottleneck resource decides to croak at 2a.m. on a
Sunday.

-- Paul

 

[Carl J. Van Arsdall]

What you describe sounds very much like pyro, which should probably be added
to your list.

Pyro looks nothing short of amazing. I will definitely spend some time
studying this technology as well. Does anyone here use pyro or any of
the other technologies enough to comment on them?

..c


--

Carl J. Van Arsdall
(e-mail address removed)
Build and Release
MontaVista Software

 

[Piet van Oostrum]

CJVA> Hey everyone, another question for the list. In particular i'm
CJVA> looking for comments on some of the distributed technologies
CJVA> supported in python. Specifically, I'm looking at XML-RPC, RPyC,
CJVA> CORBA, and Twisted.

Although it doesn't cover exactly the technologies mentioned above
the following article may be interesting for you:

Python Web services developer: Messaging technologies compared
http://www.ibm.com/developerworks/library/ws-pyth9/

 

[skip]

Piet> Python Web services developer: Messaging technologies compared
Piet> http://www.ibm.com/developerworks/library/ws-pyth9/

Note a couple things. One, the article is four years old. You can't assume
the various technologies have remained static since then. Two, the authors
apparently didn't use sgmlop (at least it's not mentioned) to boost the
decode performance of XML-RPC.

Skip

 

[yairchu]

Hey everyone, another question for the list. In particular i'm looking
for comments on some of the distributed technologies supported in
python. Specifically, I'm looking at XML-RPC, RPyC, CORBA, and Twisted.

Before you offer any comments let me talk about what i'm doing a little
bit. Right now I'm creating an execution framework. The framework aims
to allow users to create a modular system by which engineers can
mitigate some of the issues seen in large complex systems. By using
modules the system hopes to allow the system to change but allow
engineers to re-use as much code as possible. By using the modules in a
framework we hope to encapsulate the entire system to that it is
controllable in the sense that we can generate and catch errors anywhere
in the system and exit gracefully, things like that. There's a lot more
to it but the motivation spawn from our system where the system kept
needing to adapt to support changes in our company. These changes over
time have resulted in a large number of "hacks". The hope of the
modules is to make the code a lot cleaner and organized so that the
system can operate in whatever way it needs to without having so many
hacks. This system I use also operates in a primitive distributed
environment. Currently, these are ssh calls to other nodes that then
execute other python scripts and shell commands. This is the next piece
I'm interested in moving away from.

With all that being said, I've come across those four technologies so
far and I've been looking into them. I was wondering if anyone had any
comments, pros/cons, or suggestions of other distributed technologies to
take a look at. I intend to integrate the distributed technology
directly into the framework making it easy for developers to distribute
tasks. Anyhow, any comments would be greatly appreciated! I also hope
to release this framework as an open-source project when its finished,
so again I community input is important.

Thanks!

.c


--

Carl J. Van Arsdall
(e-mail address removed)
Build and Release
MontaVista Software

RPyC !
I personnaly used it (and its predecessor, pyinvoke) for:
* a testing system that controls several computers
* drawing graphs with matplotlib while avoiding its memory leaks
* distributed calculation
* controlling embedded python applications from my favorite python
shell (not the embedded one), also allowing me to control several
instances of the application.
it's so easy to use that there's was no effort require to do any of
those!

it's pretty self-explanatory, so I'll only say this:
it's Transparent. you run remote code and access remote objects just as
you would locally. it's minimal, and complete.
I highly recommend it!