Bidirectional Networking

[Emanuele D'Arrigo]

Hi everybody! A networking question!

I've been looking at and tinkering a little with the various
networking modules in python. The examples are pretty clear and a
module such as the SimpleXMLRPCServer is actually simple!

All the examples though are based on a client interrogating a server,
with the client initiating the connection, obtaining something and
then closing the connection. Basically the server is a reactive party:
only if the client get in touch the server respond. What if the server
wanted to notify the client of something of interest, i.e. new data
that the client should take into consideration and potentially
process?

One option would be for the client to periodically poll the server for
changes. Fair enough, that would work. But it'd be a bit of a waste if
the changes aren't particularly frequent.

Is it possible then to establish both a server and a client in the
same application? I guess it must be possible but the examples all
rely on some kind of server loop (i.e. SocketServer.serve_forever)
that if started on both client and server sides would create two
listening parties but no talking at all! Furthermore, other libraries
might have their own loop, i.e. a graphical client has a loop to
redraw the screen and the two loops would somehow have to be
interleaved.

I'm not quite seeing how this can be done other than we threads. Is
that the way to do it? Place the listening loop in a thread while the
rest of the application does its own thing?

Or is it SocketServer.handle_request() the key? I could call this
function periodically, but what happens if a request is made while the
application is doing something else? Are the requests queued and dealt
with one per loop?

Thanks for your help!

Manu

 

[James Mills]

Have a look at circuits.

http://trac.softcircuit.com.au/circuits/

It's a general purpose event-driven framework
with a focus on Component architectures and
has a good set of Networking Components,
specifically: circuits.lib.sockets
* TCPServer
* TCPClient
* UDPServer
* UDPClient (alias of UDPServer)

With circuits, there is no such thing as: serve_forever()
It'll try to stay out of your way as much as possible
and let you define your own main event loop which
could look like this:

from circuits import Manager
from circuits.lib.sockets import TCPServer, TCPClient

manager = Manager()
server =TCPServer(8000)
client = TCPClient()

manager += server
manager += client

....

while True:
server.poll()
client.poll()
manager.flush()

I hope this helps you!

cheers
James

 

[Tobias Andersson]

Emanuele D'Arrigo skrev:

[...] What if the server
wanted to notify the client of something of interest, i.e. new data
that the client should take into consideration and potentially
process?

If the protocol is relatively simple perhaps you can implement
something similar to IMAP's "IDLE":

http://www.faqs.org/ftp/rfc/rfc2177.txt

Basically the client (when having nothing important to do)
sends the "idle" command and then listens for messages from the
server until "idle" is canceled.

But looking into "circuits" as suggested by James seems like a
good idea also.

HTH /TA

 

[Emanuele D'Arrigo]

Thank you both for the suggestions! Eventually I tried with threading
as illustrated in the code below.
And it works pretty well! The only problem I'm having with it is that
as the server is a daemon the program should end when the client
thread cease to be alive. But it doesn't seem to work that way and I'm
not sure what's going on! I did achieve my objective though. Two
separate instances of the code below will happily send random numbers
to each other for a few seconds!

Manu

-------------------------
To use the following code, cut&paste into two separate *.py files and
invert the port numbers in one file. Then, start them in two separate
shells. WARNING: as mentioned above the two program do not exit and
must be killed, i.e. through the Windows Task Manager or the unix kill
command.
-------------------------

import SocketServer
import socket
import threading
import random
from time import sleep

## Network request handler
class MyTCPHandler(SocketServer.StreamRequestHandler):

def handle(self):
self.data = self.rfile.readline().strip()
print "-> RECV: " + self.data + " - Sent by:" +
self.client_address[0]

## Server Thread
class AsyncServer(threading.Thread):
def __init__(self, localServer):
threading.Thread.__init__(self)
self.server = SocketServer.TCPServer(localServer,
MyTCPHandler)
def run(self):
self.server.serve_forever()

## Client Thread
class AsyncClient(threading.Thread):

def __init__(self, remoteServer):
threading.Thread.__init__(self)
self.remoteServer = remoteServer

def run(self):
cycle = 0
while cycle < 1000:

chance = random.random()
if(chance < 0.01):

randomNumber = int(random.random() * 1000)
message = str(randomNumber) + " from remote cycle " +
str(cycle)

try:
sock = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
sock.connect(self.remoteServer)
sock.send(message + "\n")
sock.close()
print("SENT ->: "+str(randomNumber)+ " by local
cycle "+str(cycle))
except:
print("Failed to send number on cycle "+str
(cycle))
pass

cycle += 1

sleep(0.01)

## Simulating local/remote servers with different ports
localServer = ("localhost", 9999)
remoteServer = ("localhost", 10000)

asyncServer = AsyncServer(localServer)
asyncServer.daemon = True
asyncServer.start()

asyncClient = AsyncClient(remoteServer)
asyncClient.start()

 

[Gabriel Genellina]

Thank you both for the suggestions! Eventually I tried with threading
as illustrated in the code below.
And it works pretty well! The only problem I'm having with it is that
as the server is a daemon the program should end when the client
thread cease to be alive. But it doesn't seem to work that way and I'm
not sure what's going on! I did achieve my objective though. Two
separate instances of the code below will happily send random numbers
to each other for a few seconds!

If you're using 2.5 or older, override serve_forever:

def serve_forever(self):
while not getattr(self, 'quit', False):
self.handle_request()

and set the server 'quit' attribute to True in response to some command
from the client.
Python 2.6 has a shutdown() method for the same purpose.

 

[James Mills]

Just as a matter of completeness for my own suggestion, here
is my implementation of your code (using circuits):

cheers
James

--------------------------

import random
from circuits import listener, Event, Manager
from circuits.lib.sockets import TCPServer, TCPClient

class Server(TCPServer):

channel = "server"

@listener("read")
def onREAD(self, sock, data):
print "-> RECV: %s - Sent by: %s" % (data, hsot)

class Client(TCPClient):

channel = "client"

def __init__(self, remoteServer):
super(Client, self).__init__()
self.cycle = 0
self.randomNumber = 0
self.message = None
self.remoteServer = remoteServer

@listener("connect")
def onCONNECT(self, host, port):
self.write(self.message)
print "SENT ->: %d by by local cycle %d" % (self.randomNumber, self.cycle)
self.close()
self.success = True

@listener("disconnect")
def onDISCONNECT(self):
if not self.success:
print "Failed to send number on cycle %d " % self.cycle
self.success = None

@listener("cycle")
def onCYCLE(self):
if self.cycle < 1000:
chance = random.random()

if (chance < 0.01):
self.randomNumber = int(random.random() * 1000)
self.message = "%d from remote cycle %d" % (self.randomNumber,
self.cycle)
self.open(*self.remoteServer)
self.cycle += 1

manager.push(Event(), "cycle", "client")

localServer = (9999, "localhost")
remoteServer = ("localhost", 10000)

manager = Manager()

server = Server(*localServer)
client = Client(remoteServer)

manager += server
manager += client

manager.push(Event(), "cycle", "client")

while True:
try:
server.poll()
client.poll()
manager.flush()
except KeyboardInterrupt:
client.close()
break

 

[Bryan Olson]

All the examples though are based on a client interrogating a server,
with the client initiating the connection, obtaining something and
then closing the connection. Basically the server is a reactive party:
only if the client get in touch the server respond.

Indeed, to the network programmer that pattern of interaction defines
client/server (according to Douglas Comer's /Internetworking with TCP/IP).

What if the server
wanted to notify the client of something of interest, i.e. new data
that the client should take into consideration and potentially
process?

The "server push" problem.

One option would be for the client to periodically poll the server for
changes. Fair enough, that would work. But it'd be a bit of a waste if
the changes aren't particularly frequent.

Implementing server push on top of client poll is the most popular and
generally simplest option.

Is it possible then to establish both a server and a client in the
same application?

Possible, and not all that hard to program, but there's a gotcha.
Firewalls, including home routers and software firewalls, typically
default to disallowing connections in the 'wrong' direction. If the
client initiates all connections, you avoid a world of hassles.

I guess it must be possible but the examples all
rely on some kind of server loop (i.e. SocketServer.serve_forever)
that if started on both client and server sides would create two
listening parties but no talking at all! Furthermore, other libraries
might have their own loop, i.e. a graphical client has a loop to
redraw the screen and the two loops would somehow have to be
interleaved.

I'm not quite seeing how this can be done other than we threads. Is
that the way to do it? Place the listening loop in a thread while the
rest of the application does its own thing?

Threads work. Other options in Python standard library are 'select' and
'asyncore', but these do not play well with SocketServer and its
descendants. There are also other free Python libraries.

Or is it SocketServer.handle_request() the key? I could call this
function periodically, but what happens if a request is made while the
application is doing something else? Are the requests queued and dealt
with one per loop?

Requests are queued up, but the queue is small. SocketServer is built on
top of 'socket', which has the list() and accept() calls to create the
queue and get connections from it, respectively. The traditional listen
queue size, and sometimes the system maximum, is 5.

If you want to use SocketServer, read about ThreadingMixIn and ForkingMixIn.

 

[Emanuele D'Arrigo]

If you're using 2.5 or older, override serve_forever:

def serve_forever(self):
while not getattr(self, 'quit', False):
self.handle_request()

and set the server 'quit' attribute to True in response to some command
from the client.

Ok, I've tried this method and it would work if I wanted to shut down
the server from the remote client. But if I want the server to shut
down from the server itself upon some condition it doesn't work
because the while statement is evaluated again only after there has
been a request. If requests are infrequent or have ceased the
self.handle_request() is never invoked and both the loop and the
thread that contains it hang. I wasn't able to set a timeout upon
which the handle_request() method returns. I'm not sure why. But I was
able to make the original serve_forever() work.

The problem was that in my code I used:

asyncServer.daemon = True

but somehow that doesn't work. I then tried:

asyncServer.setDaemon(True)

and that does work: when the asyncClient thread ends the only thread
left running is the asyncServer, but as it is a daemon thread and
there are no non-daemon threads left, the application exits regularly.
So, below there's an updated version of my code that works as
intended.

Thanks for your help!

Manu


-------------------------
To use the following code, cut&paste into two separate *.py files and
invert the port numbers in one file. Then, start them in two separate
shells.
-------------------------

import SocketServer
import socket
import threading
import random
from time import sleep

## Network request handler
class MyTCPHandler(SocketServer.StreamRequestHandler):

def handle(self):
self.data = self.rfile.readline().strip()
print "-> RECV: " + self.data + " - Sent by:" +
self.client_address[0]

## Server Thread
class ServerThread(threading.Thread):
def __init__(self, localServer):
threading.Thread.__init__(self)
self.server = SocketServer.TCPServer(localServer,
MyTCPHandler)

def run(self):
self.server.serve_forever()

## Client Thread
class ClientThread(threading.Thread):

def __init__(self, remoteServer):
threading.Thread.__init__(self)
self.remoteServer = remoteServer

def run(self):
cycle = 0
while cycle < 1000:

chance = random.random()
if(chance < 0.01):

randomNumber = int(random.random() * 1000)
message = str(randomNumber) + " from remote cycle " +
str(cycle)

try:
sock = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
sock.connect(self.remoteServer)
sock.send(message + "\n")
sock.close()
print("SENT ->: "+str(randomNumber)+ " by local
cycle "+str(cycle))
except:
print("Failed to send number on cycle "+str
(cycle))

cycle += 1
sleep(0.01)

## Simulating local/remote servers with different ports
localServer = ("localhost", 9999)
remoteServer = ("localhost", 10000)

serverThread = ServerThread(localServer)
serverThread.setDaemon(True)
serverThread.start()

clientThread = ClientThread(remoteServer)
clientThread.start()


---------------------------

 

[Emanuele D'Arrigo]

Just as a matter of completeness for my own suggestion, here
is my implementation of your code (using circuits):

It's longer! But I bet is a little bit more resilient against all
sorts of problems that arise while using network connections.

Well, thank you for that. The code I posted in this thread it was just
a test to understand how it all works and how it all works with python
out of the box. When I get to write the actual application I might end
up using circuits instead!

Thank you again!

Manu

 

[Emanuele D'Arrigo]

Hey Bryan, thank you for your reply!

Possible, and not all that hard to program, but there's a gotcha.
Firewalls, including home routers and software firewalls, typically
default to disallowing connections in the 'wrong' direction. If the
client initiates all connections, you avoid a world of hassles.

Ah yes, I can see that. Uhm. I have absolutely no idea right now how a
firewall works from a programming point of view and what happens in
normal "residential" circumstances. I.e. it's clear that firewalls are
configured to allow http traffic because I can browse the internet. Is
that done leaving a specific port open? Or does the browser request
the firewall to open a specific port for it and the firewall trust the
browser to handle safely anything that comes through?

I.e. in the case of the code in this thread, would it be the
responsibility of the application to tunnel through the firewall and
listen for connections or would it be the responsibility of the user
to configure the firewall so that the application can receive a
connection?

Thanks for your help!

Manu

 

[Bryan Olson]

Hey Bryan, thank you for your reply!



Ah yes, I can see that. Uhm. I have absolutely no idea right now how a
firewall works from a programming point of view and what happens in
normal "residential" circumstances. I.e. it's clear that firewalls are
configured to allow http traffic because I can browse the internet. Is
that done leaving a specific port open? Or does the browser request
the firewall to open a specific port for it and the firewall trust the
browser to handle safely anything that comes through?

Software firewalls will often simply refuse incoming connections. The
basic protection of the garden-variety home router comes from "network
address translation" (NAT), in which case TCP connections initiated from
the inside will generally work, regardless of port, and incoming
connections will fail.

Internet server farms often enforce the other side of the client-side
policy, with firewalls configured to disallow outgoing initiation of
connections.

If the application need to work in restrictive environments where
firewalls only pass known protocols, a popular approach to build the
application protocol on top of HTTP, with all the required standard
headers and a new content-type.

I.e. in the case of the code in this thread, would it be the
responsibility of the application to tunnel through the firewall and
listen for connections

I'm not clear on what that means.

or would it be the responsibility of the user
to configure the firewall so that the application can receive a
connection?

That can be a huge hassle. The first choice is for the application to
conform to popular firewall policies, so no special configuration is
required.

 

[Emanuele D'Arrigo]

Software firewalls will often simply refuse incoming connections. The
basic protection of the garden-variety home router comes from "network
address translation" (NAT), in which case TCP connections initiated from
the inside will generally work, regardless of port, and incoming
connections will fail.

Ok, I think I'm getting the picture here. So this means that in most
circumstances where the data flow from the server is frequent the
client initiates the connection, usually requests some initial data
and keeps polling the server periodically, issuing new requests. In
this context can the client simply keep the connection alive and
listen for new data from the server coming at any time rather than
actively issuing requests? Are there drawbacks to this strategy? I.e.
is there a limit to the number of simultaneous connections a server
can keep alive? I've noticed that the socket pages mention a 5
connections limit. Is that it? What if I want to make a virtual room
with 20 people connected simultaneously?

That can be a huge hassle. The first choice is for the application to
conform to popular firewall policies, so no special configuration is
required.

I agree, I'd rather have the user do nothing in this regard. I'm just
wondering how it's done with data intensive application where the
server needs to send new data to the client frequently. Does the
client just keep the connection live at all time for the server to
send stuff or does the client continuously open, sends a request,
receives data and closes the connection every time? Here I'm thinking
about an online game, with 100 players moving their avatars. Does the
client requests their position nearly every frame?

Manu

 

[Brian Allen Vanderburg II]

Ok, I think I'm getting the picture here. So this means that in most
circumstances where the data flow from the server is frequent the
client initiates the connection, usually requests some initial data
and keeps polling the server periodically, issuing new requests. In
this context can the client simply keep the connection alive and
listen for new data from the server coming at any time rather than
actively issuing requests? Are there drawbacks to this strategy? I.e.
is there a limit to the number of simultaneous connections a server
can keep alive? I've noticed that the socket pages mention a 5
connections limit. Is that it? What if I want to make a virtual room
with 20 people connected simultaneously?

I've done some network programming not much. I think if you need to
receive update from a server frequently a constant connection would be
better than connect-request-disconnect. As for the backlog (5), this
doesn't mean that you can only have a maximum of 5 established
connections. Each established connection gets a new socket object. But
what I think it means is that during the listen for an incoming
connection on the listening socket, if multiple connection attempts are
coming in at one time it can keep a backlog of up to 5 of these
connection attempts for that individual socket.


Brian Vanderburg II

 

[Gabriel Genellina]

Ok, I've tried this method and it would work if I wanted to shut down
the server from the remote client. But if I want the server to shut
down from the server itself upon some condition it doesn't work
because the while statement is evaluated again only after there has
been a request. If requests are infrequent or have ceased the

Yes, this is the problem with this simple approach, handle_request is
blocking. Python 2.6 has great improvements on this situation.

The problem was that in my code I used:

asyncServer.daemon = True

but somehow that doesn't work. I then tried:

asyncServer.setDaemon(True)

and that does work:

daemon became a property in Python 2.6; setDaemon was the only way to set
it in previous versions.

Thanks for your help!

Thanks for sharing your code!

 

[Emanuele D'Arrigo]

daemon became a property in Python 2.6; setDaemon was the only way to set
it in previous versions.

I thought that might be the case! The documentation is a bit vague:

http://docs.python.org/library/threading.html?highlight=threading#threading.Thread.setDaemon

Should we maybe add "New in 2.6" to the Thread.daemon paragraph?

Manu

 

[Emanuele D'Arrigo]

But what I think it means is that during the listen for an incoming
connection on the listening socket, if multiple connection attempts are
coming in at one time it can keep a backlog of up to 5 of these
connection attempts for that individual socket.

Ah, Gotcha! Thank you, that makes sense!

Manu

 

[Gabriel Genellina]

I thought that might be the case! The documentation is a bit vague:

http://docs.python.org/library/threading.html?highlight=threading#threading.Thread.setDaemon

Should we maybe add "New in 2.6" to the Thread.daemon paragraph?

That would be a nice addition, but if you are using Python 2.5, you should
read the 2.5 version of the docs:
http://www.python.org/doc/2.5.2/lib/thread-objects.html

 

[Bryan Olson]

Ok, I think I'm getting the picture here. So this means that in most
circumstances where the data flow from the server is frequent the
client initiates the connection, usually requests some initial data
and keeps polling the server periodically, issuing new requests. In
this context can the client simply keep the connection alive and
listen for new data from the server coming at any time rather than
actively issuing requests?

Sure. You might include a method for the client to ensure the connection
is alive, either with TCP keep-alive or your own messages over the
connection.

Are there drawbacks to this strategy? I.e.
is there a limit to the number of simultaneous connections a server
can keep alive? I've noticed that the socket pages mention a 5
connections limit. Is that it? What if I want to make a virtual room
with 20 people connected simultaneously?

Five is not the connection limit. Twenty connections is no problem, and
modern systems should support hundreds by default, and thousands with a
bit of care and tuning.

I agree, I'd rather have the user do nothing in this regard. I'm just
wondering how it's done with data intensive application where the
server needs to send new data to the client frequently. Does the
client just keep the connection live at all time for the server to
send stuff or does the client continuously open, sends a request,
receives data and closes the connection every time? Here I'm thinking
about an online game, with 100 players moving their avatars. Does the
client requests their position nearly every frame?

I'd say keeping an open connection to each client is reasonable there.
SocketServer and its descendants may not be sophisticated enough for
such an application.

 

[Bryan Olson]

As for the backlog (5), this
doesn't mean that you can only have a maximum of 5 established
connections. Each established connection gets a new socket object. But
what I think it means is that during the listen for an incoming
connection on the listening socket, if multiple connection attempts are
coming in at one time it can keep a backlog of up to 5 of these
connection attempts for that individual socket.

Right. An incoming connect adds a connection to the listen queue, and
the application's accept() call removes one connections from the queue.

The limited backlog is relevant to the OP's suggestion of calling
SocketServer.handle_request() periodically. The accept() that pulls
connections off the listen queue is within handle_request(), thus during
the period between these calls incoming connections will queue up and
could easily reach the limit.

 

[Emanuele D'Arrigo]

Thanks everybody and in particular Gabriel and Bryan for their
contributions to this thread. Very much useful information.

Manu