Non blocking socket server and storage engine

[kdeveloper]

Hello Pythonists,

I am building a non blocking socket server for incomming UDP packets.
The server needs to run at least three threads:
1. getting data and pushing to "some" storage (at the moment I use
queue),
2. acknowledge the package received
3. retrieve the information from the storage and insert it in DB.

The problem I have is that when I use queue it stores more packets in
the queue than it actually receives. An example: sent 99 UDP packets
and queue stored 600-750 entries (?) Why? I have no idea. I have
impression that I still do not understand completely how does the
queue work in python.

Another issue is that I want the server to run very rupidly. The
server needs to get at least 100 packets per second, and the same
amount of acknowledges has to be sent back.

The questions are:
1. What is the best storage engine for Python and multithreading
server?
2. Can I use queue for such problem?
3. How does actually queue work in Python? (I know how it should work
generally, but somehow it doesn't work as I expect)

Any hints & helps? Would be very grateful

Cheers
K

 

[Lawrence D'Oliveiro]

The server needs to run at least three threads:

Get it working without threading first.

 

[Chris Rebert]

Hello Pythonists,

I am building a non blocking socket server for incomming UDP packets.
The server needs to run at least three threads:
1. getting data and pushing to "some" storage (at the moment I use
queue),
2. acknowledge the package received
3. retrieve the information from the storage and insert it in DB.

The problem I have is that when I use queue it stores more packets in
the queue than it actually receives. An example: sent 99 UDP packets
and queue stored 600-750 entries (?) Why? I have no idea. I have
impression that I still do not understand completely how does the
queue work in python.

No, I believe rather you don't completely understand UDP.

Quoting from Wikipedia (http://en.wikipedia.org/wiki/User_Datagram_Protocol):
"UDP does not guarantee reliability or ordering in the way that TCP
does. Datagrams may arrive out of order, ***appear duplicated***, or
go missing without notice."

I think that might at least partially account for your duplicate entries.

Cheers,
Chris

 

[kdeveloper]

Quoting from Wikipedia (http://en.wikipedia.org/wiki/User_Datagram_Protocol):
"UDP does not guarantee reliability or ordering in the way that TCP
does. Datagrams may arrive out of order, ***appear duplicated***, or
go missing without notice."

I agree with that. But the amount of possible duplcates shouldn't be
so huge.

What about using different storing engine? Any suggestions?

Cheers
K

 

[kdeveloper]

Get it working without threading first.

Lawrence, I was following article [1] when building my code.
So I guess it should be ok. At the moment
I have only two threads running:
a. receiving packages and storing to queue,
b. getting packet from the queue and storing in DB.

Regards,
Krzysztof


[1] http://www.ibm.com/developerworks/aix/library/au-threadingpython/index.html