Need some IPC pointers

[Andrew Berg]

I've done some research, but I'm not sure what's most appropriate for my
situation. What I want to do is have a long running process that spawns
processes (that aren't necessarily written in Python) and communicates
with them. The children can be spawned at any time and communicate at
any time. Being able to communicate with non-local processes would be
nice, but is not necessary. The implementation needs to be
cross-platform, but child processes will use the same OS as the parent
during runtime.
I don't think I'll ever need to transfer anything complicated or large -
just strings or possibly tuples/lists. I'd rather not go outside the
standard library (but I'd consider it). I don't need to worry about
compatibility with older Python versions; if it only works with Python
3.2, that's not a problem.
I'm thinking sockets, but perhaps there's something simpler/easier.

 

[Miki Tebeka]

There are two different problems. One is the medium to pass messages, sockets are good but there are other options and depends on your requirement you need to pick the best one.
The other is serialization format, here you have marshal, pickle, JSON, XML and more.

For me JSON over sockets works well in the past. It has the nice property that you can read messages without need for a special decoder (like in the case of marshal/pickle).

 

[Miki Tebeka]

There are two different problems. One is the medium to pass messages, sockets are good but there are other options and depends on your requirement you need to pick the best one.
The other is serialization format, here you have marshal, pickle, JSON, XML and more.

For me JSON over sockets works well in the past. It has the nice property that you can read messages without need for a special decoder (like in the case of marshal/pickle).

 

[Irmen de Jong]

I've done some research, but I'm not sure what's most appropriate for my
situation. What I want to do is have a long running process that spawns
processes (that aren't necessarily written in Python) and communicates
with them. The children can be spawned at any time and communicate at
any time. Being able to communicate with non-local processes would be
nice, but is not necessary. The implementation needs to be
cross-platform, but child processes will use the same OS as the parent
during runtime.
I don't think I'll ever need to transfer anything complicated or large -
just strings or possibly tuples/lists. I'd rather not go outside the
standard library (but I'd consider it). I don't need to worry about
compatibility with older Python versions; if it only works with Python
3.2, that's not a problem.
I'm thinking sockets, but perhaps there's something simpler/easier.

Standard library, local processes: multiprocessing module.

If that doesn't suit your needs, maybe check out Pyro:
http://packages.python.org/Pyro4/

Pyro allows objects to talk to each other over the network, with minimal
programming effort. You can just use normal Python method calls to call
objects on other machines (or locally, ofcourse). It's written in 100%
pure Python and works on Python 2.6 and upwards (including 3.x).

Regards,
Irmen de Jong

 

[Irmen de Jong]

On 30-11-11 22:03, Andrew Berg wrote:

Oops, missed this on my first read. This rules out my suggestion of Pyro
because that requires Python on both ends (or Java/.net on the client
side). Sorry for the noise.

Irmen

 

[bobicanprogram]

I've done some research, but I'm not sure what's most appropriate for my
situation. What I want to do is have a long running process that spawns
processes (that aren't necessarily written in Python) and communicates
with them. The children can be spawned at any time and communicate at
any time. Being able to communicate with non-local processes would be
nice, but is not necessary. The implementation needs to be
cross-platform, but child processes will use the same OS as the parent
during runtime.
I don't think I'll ever need to transfer anything complicated or large -
just strings or possibly tuples/lists. I'd rather not go outside the
standard library (but I'd consider it). I don't need to worry about
compatibility with older Python versions; if it only works with Python
3.2, that's not a problem.
I'm thinking sockets, but perhaps there's something simpler/easier.

You might want to take a look at the SIMPL toolkit.
http://www.icanprogram.com/06py/lesson1/lesson1.html

SIMPL modules can be written (and mixed) in any number of supported
languages including Python, C, C++, JAVA, Tcl/Tk or PHP. Non local
communication is handled by generic surrogates. Because of this
surrogate architecture SIMPL modules can often be tested locally and
then deployed into the cloud without any changes.

bob

 

[Floris Bruynooghe]

I'm surprised no one has mentioned zeromq as transport yet. It provides scaling from in proc (between threads) to inter-process and remote machines in a fairly transparent way. It's obviously not the python stdlib and as any system there are downsides too.

Regards,
Floris

 

[Floris Bruynooghe]

I'm surprised no one has mentioned zeromq as transport yet. It provides scaling from in proc (between threads) to inter-process and remote machines in a fairly transparent way. It's obviously not the python stdlib and as any system there are downsides too.

Regards,
Floris

 

[Andrew Berg]

What about named pipes? I don't mind a bit of "if Windows do this, else,
do that" as long I'm not coding two or more completely different
approaches. I'm not too familiar with named pipes, though; perhaps
someone with some experience could chime in.


Apparently this didn't go through to Google Groups the first time; it
may show up twice for some.