Distributed multitasking, POE, communication

[schwarzenschafe]

I'm writing a distributed WWW media (images) robot. I want a central
server that holds the database and manages the other servers. These
other servers will manage several robot processes. Every server will
have different access to the web, and must communicate over the
internet.

I'd like to use POE, I need some kind of event driven management of
each server. I can't run the robots themselves in POE because they'll
be waiting for LWP to return webpages, and I'd also like to sleep()
which makes the whole POE driven server (all robots) stop.

The best setup I can come up with uses domain sockets for the robot
proccesses to communicate with the POE management process, which itself
can communicate with the POE central server. I'm not very clear on how
the servers will communicate. I want a permenant bidirectional
connection, but I don't know if I can make a single TCP/IP connection
with POE and have either server send data to the other to trigger
events on the remote server. Likewise if the robot process and POE can
monitor local sockets for bidirectional communication on teh same
machine. Any help is appreciated, including which modules I should look
at. I've found a lot of books on robots, read some. And yes, I know
what a monumental undertaking this is (I either have a lot of time or
I'm a senseless dreamer, or both). Regards,

- SS

 

[Ben Morrow]

[disclaimer: I have never used POE. The below is based on the
documentation]

Quoth (e-mail address removed):

I'm writing a distributed WWW media (images) robot. I want a central
server that holds the database and manages the other servers. These
other servers will manage several robot processes. Every server will
have different access to the web, and must communicate over the
internet.

I'd like to use POE, I need some kind of event driven management of
each server. I can't run the robots themselves in POE because they'll
be waiting for LWP to return webpages,

You can use POE::Component::Client::HTTP to get webpages, instead of
LWP. Then you can integrate the fetching into your manager process,
which will probably be more efficient.

and I'd also like to sleep()
which makes the whole POE driven server (all robots) stop.

Well, yes. Why do you think you want to sleep? You can use
POE::Kernel->delay_set to be notified after a certain amount of time has
passed.

The best setup I can come up with uses domain sockets

ITYM 'Unix-domain sockets'? Every socket has a domain (usually called
the 'address family'); the point about Unix-domain sockets is that the
domain is the local Unix machine rather than the Internet.

for the robot
proccesses to communicate with the POE management process, which itself
can communicate with the POE central server. I'm not very clear on how
the servers will communicate. I want a permenant bidirectional
connection, but I don't know if I can make a single TCP/IP connection
with POE and have either server send data to the other to trigger
events on the remote server.

I don't quite understand what you mean here... are you talking about the
slave servers communicating with each other, or only with the central
master server?

The way I would structure this would be to have each slave process (your
'POE management process') connect to a listening socket on the master
process (your 'POE central server'), and then communicate with it by
some protocol you define. It is perfectly possible to define the
protocol such that either end can initiate transactions: you just have
to be careful to make sure both ends don't try to speak at the same time
so the transactions get mixed up. It's worth putting some though into
the design of the protocol before you start programming: IMO HTTP/1.1 is
a good model to follow, but your situation is somewhat complicated by
the fact the either end can speak first.

Likewise if the robot process and POE can
monitor local sockets for bidirectional communication on teh same
machine.

Well, yes. That's the point: to watch several filehandles for activity.
Whether the filehandles are AF_INET sockets, AF_UNIX sockets or whatever
doesn't matter: that's the point of the Unix filehandle abstraction.

Any help is appreciated, including which modules I should look
at. I've found a lot of books on robots, read some. And yes, I know
what a monumental undertaking this is (I either have a lot of time or
I'm a senseless dreamer, or both).

Heh. Good luck.

Ben

 

[schwarzenschafe]

The way I would structure this would be to have each slave process (your
'POE management process') connect to a listening socket on the master
process (your 'POE central server'), and then communicate with it by
some protocol you define. It is perfectly possible to define the
protocol such that either end can initiate transactions: you just have
to be careful to make sure both ends don't try to speak at the same time
so the transactions get mixed up. It's worth putting some though into
the design of the protocol before you start programming: IMO HTTP/1.1 is
a good model to follow, but your situation is somewhat complicated by
the fact the either end can speak first.

I've been talking with Uri Guttman and he has me sold on Stem
(http://www.stemsystems.com), which will do this for me. I'm also
rethinking my strategy for managing the different servers, I have alot
of planning still. Thanks for the response!

SS