socket problem

[Gordon Wetzstein]

Hello everyone,

I have a problem with python sockets.
I broadcast a lot of pickled objects with socket. sendto(...), that works.
Ireceive them on the other site with socket.recvfrom(16384)
The pickled objects are smaller (like 10000 bytes) than the max bytes.

The problem appears if I send too many pickled objects very quickly one
after another,
then I only receive a part of the sent objects.
I print them before I send them, they are there but they never reach the
destination.
When I do a time.sleep(0.1) (but not lesser than 0.1) after socket.send()
all is good but it's too slow!

Does anyone know what the problem is? And maybe a solution!

thanks gordon

 

[Alan Kennedy]

I have a problem with python sockets.
I broadcast a lot of pickled objects with socket. sendto(...), that
works.
Ireceive them on the other site with socket.recvfrom(16384)
The pickled objects are smaller (like 10000 bytes) than the max bytes.

The problem appears if I send too many pickled objects very quickly one
after another,
then I only receive a part of the sent objects.

As Jp Calderone pointed out, UDP is an unreliable protocol. Therefore,
it *will* drop packets occasionally. Worse, it doesn't guarantee
ordering of delivery of packets, meaning that your pickles could
arrive all jumbled up and unusable.

You have three main choices to get around this problem.

1. Use TCP, which guarantees delivery to the destination address once
and only once, guarantees that packets will be received in the order
in which they were sent. However, you can't broadcast with TCP: it's a
point to point protocol.

2. Implement your own guaranteed delivery protocol over UDP. This is
usually done by assigning packets a sequence number as they are
transmitted from the publisher. All consumers then keep track of the
sequence numbers they receive, and re-request any packets they missed.
Implementing your own such protocol can get quite tricky, depending on
how complex your requirements.

3. Use a pre-made, highly efficient python library which is custom
designed for the job: spread. Spread is designed to solve all of the
problems of reliable broadcasting over UDP, and gives you a wide range
of options for how to balance efficiency versus reliability. And more
importantly, it's industrial-strength robust, stable and platform
independent. Also, it's configurable to work with a wide range of
network topologies. More info from

http://www.python.org/other/spread/

If I were you, I'd seriously consider spending an hour getting up and
running with Spread: could be the most productive hour you'll spend in
this area.

HTH,

 

[Gordon Wetzstein]

As Jp Calderone pointed out, UDP is an unreliable protocol. Therefore,
it *will* drop packets occasionally. Worse, it doesn't guarantee
ordering of delivery of packets, meaning that your pickles could
arrive all jumbled up and unusable.

You have three main choices to get around this problem.

1. Use TCP, which guarantees delivery to the destination address once
and only once, guarantees that packets will be received in the order
in which they were sent. However, you can't broadcast with TCP: it's a
point to point protocol.

2. Implement your own guaranteed delivery protocol over UDP. This is
usually done by assigning packets a sequence number as they are
transmitted from the publisher. All consumers then keep track of the
sequence numbers they receive, and re-request any packets they missed.
Implementing your own such protocol can get quite tricky, depending on
how complex your requirements.

3. Use a pre-made, highly efficient python library which is custom
designed for the job: spread. Spread is designed to solve all of the
problems of reliable broadcasting over UDP, and gives you a wide range
of options for how to balance efficiency versus reliability. And more
importantly, it's industrial-strength robust, stable and platform
independent. Also, it's configurable to work with a wide range of
network topologies. More info from

http://www.python.org/other/spread/

If I were you, I'd seriously consider spending an hour getting up and
running with Spread: could be the most productive hour you'll spend in
this area.

HTH,

thanks, i will try it.
gordon