Problem receiving UDP broadcast packets.

[Grant Edwards]

I'm have problems figuring out how to receive UDP broadcast packets on
Linux.

Here's the receiving code:

------------------------------receive.py-------------------------------
#!/usr/bin/python
import socket

host = ''
port = 5010

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host, port))

while 1:
try:
message = s.recv(8192)
print "Got data: %s" % repr(message)
except KeyboardInterrupt:
break
----------------------------------------------------------------------

Here's the sending code:

--------------------------------send.py-------------------------------
#!/usr/bin/python
import sys,socket,time

host = sys.argv[1]
port = 5010

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host,port))

s.sendto(str(time.time()), ('255.255.255.255', port))
----------------------------------------------------------------------


On the receiving machine, I've used tcpdump to verify that broadcast
packets are being seen and have a destination IP of 255.255.255.255 and
destination MAC of ff:ff:ff:ff:ff:ff

03:05:09.187327 IP 10.0.0.1.5010 > 255.255.255.255.5010: UDP, length 13
0x0000: ffff ffff ffff 0018 e708 2033 0800 4500
0x0010: 0029 0000 4000 4011 30c4 0a00 0001 ffff
0x0020: ffff 1392 1392 0015 6e6e 3133 3033 3235
0x0030: 3131 3830 2e34 3500 0000
03:05:09.407508 IP 10.0.0.1.5010 > 255.255.255.255.5010: UDP, length 13
0x0000: ffff ffff ffff 0018 e708 2033 0800 4500
0x0010: 0029 0000 4000 4011 30c4 0a00 0001 ffff
0x0020: ffff 1392 1392 0015 6c6c 3133 3033 3235
0x0030: 3131 3830 2e36 3700 0000
03:05:09.615962 IP 10.0.0.1.5010 > 255.255.255.255.5010: UDP, length 13
0x0000: ffff ffff ffff 0018 e708 2033 0800 4500
0x0010: 0029 0000 4000 4011 30c4 0a00 0001 ffff
0x0020: ffff 1392 1392 0015 6b6a 3133 3033 3235
0x0030: 3131 3830 2e38 3800 0000

But, the receiving Python program never sees any packets unless the
_source_ IP address in the packets is on the same subnet as the
receiving machine. In this test case, the receiving machine has an IP
address of 172.16.12.34/16. If I change the receiving machine's IP
address to 10.0.0.123, then the receiving program sees the packets.

Even though the destination address is 255.255.255.255, the receiving
machine appears to discard the packets based on the _source_ IP. Can
anybody provide example Python code for Linux that receives UDP
broadcast packets regardless of their source IP address?

This probably is more of a Linux networking question than a Python
question, but I'm hoping somebody has solved this problem in Python.

 

[Irmen de Jong]

I'm have problems figuring out how to receive UDP broadcast packets on
Linux.
[...]

Here's the sending code:

--------------------------------send.py-------------------------------
#!/usr/bin/python
import sys,socket,time

host = sys.argv[1]
port = 5010

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host,port))

I don't think you should use s.bind() at all in the sending code.
Could that be at least part of the problem?


-Irmen

 

[Grant Edwards]

I'm have problems figuring out how to receive UDP broadcast packets on
Linux.
[...]

Here's the sending code:

--------------------------------send.py-------------------------------
#!/usr/bin/python
import sys,socket,time

host = sys.argv[1]
port = 5010

s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_BROADCAST, 1)
s.bind((host,port))

I don't think you should use s.bind() at all in the sending code.
Could that be at least part of the problem?

If I don't call bind(), then the broadcast packets go out the wrong
interface on the sending machine.

 

[Dan Stromberg]

I'm have problems figuring out how to receive UDP broadcast packets on
Linux.

Here's the receiving code:

------------------------------receive.py-------------------------------

But, the receiving Python program never sees any packets unless the
_source_ IP address in the packets is on the same subnet as the
receiving machine.

This is just how broadcasts work.

Normally, you take your subnet mask and "bitwise and" it with the IP
addresses of the sending and receiving machines. If the results match
for two such pairs on two different machines, then the broadcast
should be visible, given appropriate code.

However, some routers have the ability to pass packets from one subnet
to another. I believe this is called a "helper", at least in the
Cisco world, and must be configured specially.

 

[Irmen de Jong]

If I don't call bind(), then the broadcast packets go out the wrong
interface on the sending machine.

Fair enough.

Next issue then: as far as I know, broadcast packets are by default not routed across
subnets by gateways. Which is a good thing.

That would explain why your receiver doesn't see the packets unless its interface IP
address is in the same subnet as the sender's.

However it doesn't explain (for me) why the tcpdump program running on that same
receiver machine still happily spits out received packets. Unless the routing between
the subnets is somehow done on the receiving machine itself? My knowledge of networks
and TCP/IP ends here I'm afraid.

Cheers
Irmen.

 

[Dan Stromberg]

Fair enough.

Next issue then: as far as I know, broadcast packets are by default not routed across
subnets by gateways. Which is a good thing.

That would explain why your receiver doesn't see the packets unless its interface IP
address is in the same subnet as the sender's.

However it doesn't explain (for me) why the tcpdump program running on that same
receiver machine still happily spits out received packets. Unless the routing between
the subnets is somehow done on the receiving machine itself? My knowledge of networks
and TCP/IP ends here I'm afraid.

Cheers
Irmen.

I'm guessing there are two different subnets on the same physical
cable - which is a little unusual, but not impossible.

 

[Grant Edwards]

Fair enough.

Next issue then: as far as I know, broadcast packets are by default
not routed across subnets by gateways. Which is a good thing.

That would explain why your receiver doesn't see the packets unless
its interface IP address is in the same subnet as the sender's.

However it doesn't explain (for me) why the tcpdump program running
on that same receiver machine still happily spits out received
packets.

The two machines are on the same Ethernet segment (they're connected
via a dumb Ethernet switch).

Tcpdump shows the packets because the packets are being received by
the receiving machine's Ethernet interface. They have a destination
MAC of ff:ff:ff:ff:ff:ff, so everybody on the logical Ethernet segment
receives them.

I guess the problem is that I expected to receive a packet on an
interface anytime a packet was received with a destination IP address
that matched that of the the interface. Apprently there's some
filtering in the network stack based on the _source_ address as well
(that seems very counter-intuitive to me).

 

[Grant Edwards]

This is just how broadcasts work.

Normally, you take your subnet mask and "bitwise and" it with the IP
addresses of the sending and receiving machines. If the results match
for two such pairs on two different machines, then the broadcast
should be visible, given appropriate code.

That certainly looks like what's hapenning, but it seems very
counter-productive to me.

If I send a packet to ff:ff:ff:ff:ff:ff--255.255.255.255, it's because
I want everybody on the Ethernet segment to receive it. If I wanted
only people on a particular subnet (e.g. 10.0.0.0/8) to receive it, I
would have sent it to the subnet broadcast address (e.g. 10.255.255.255).

However, some routers have the ability to pass packets from one subnet
to another. I believe this is called a "helper", at least in the
Cisco world, and must be configured specially.

There are no routers or firewalls involved -- just a dumb Ethernet
switch.

It seems I'm going to have to use raw sockets to do what I need to do.
That's exactly what I was trying to avoid by using UDP: I'm replacing
a proprietary (non IP) MAC-level protocol that was implemented using
raw sockets.

 

[Grant Edwards]

I'm guessing there are two different subnets on the same physical
cable

Yes -- though technically they're on the same Ethernet segment rather
than physical cable, since there's an intervening Ethernet switch.

- which is a little unusual, but not impossible.

OK, here's some background...

I'm trying to implement a device discovery/configuration protocol that
uses UDP broadcast packets to discover specific types of devices on
the local Ethernet segment. The management program broadcasts a
discovery command to a particular UDP port. All devices who get that
packet are expected to answer regardless of thier current IP address.

The management program can then send another broadcast packet to
configure the IP address of a device. After that, the management
program switches over to normal unicast TCP and UDP protocols (HTTP,
TFTP, etc.) to set up the device.

I had ignorantly assumed that an UDP broadcast sent to IP address
255.255.255.255 would be received by everybody who could hear it.

Apparently I'm going to have to use RAW packets and implement UDP
myself. :/

 

[Chris Angelico]

If I send a packet to ff:ff:ff:ff:ff:ff--255.255.255.255, it's because
I want everybody on the Ethernet segment to receive it.  If I wanted
only people on a particular subnet (e.g. 10.0.0.0/8) to receive it, I
would have sent it to the subnet broadcast address (e.g. 10.255.255.255).

It seems I'm going to have to use raw sockets to do what I need to do.
That's exactly what I was trying to avoid by using UDP: I'm replacing
a proprietary (non IP) MAC-level protocol that was implemented using
raw sockets.

I have to ask:

1) Why do you have two subnets on the same switch? Isn't that going to
be an eternal maintenance headache? Not that it _can't_ be done -
obviously it can - but it's likely to confuse the humans involved.

2) Can you replace that protocol at a higher level? Presumably there's
a full protocol stack with application data getting wrapped up inside
(ultimately) ethernet frames; can you cut it somewhere else and make,
say, a TCP/IP connection to the remote system?

Chris Angelico

 

[Chris Angelico]

The management program can then send another broadcast packet to
configure the IP address of a device. After that, the management
program switches over to normal unicast TCP and UDP protocols (HTTP,
TFTP, etc.) to set up the device.

Wonder if it would be possible to (ab)use DHCP for this. If every
device registers itself with a central DHCP server, you could query
that to find out what's around, and configuring of IP addresses would
then be out of your hands.

Or can you simply use a stupid netmask like /1 that picks up all the
IP ranges? That way, the source-IP check wouldn't fail.

Chris Angelico

 

[Grant Edwards]

Wonder if it would be possible to (ab)use DHCP for this. If every
device registers itself with a central DHCP server, you could query
that to find out what's around, and configuring of IP addresses would
then be out of your hands.

I can't require the presense of a DHCP server, and many installations
won't have one.

Or can you simply use a stupid netmask like /1 that picks up all the
IP ranges? That way, the source-IP check wouldn't fail.

That would require that the device somehow knows that it's not
configured correctly and should change the netmask to /1. The device
doesn't have any way to know that, and it must respond to the
discovery commands both before and after it's properly configured.

I've reread the protocol documentation and noticed that the device has
to respond not only to broadcasts to 255.255.255.255 but also to
subnet broadcasts send to subnets it's not on. That pretty much
clinches the requirement to use a raw socket. :/

 

[Chris Angelico]

That would require that the device somehow knows that it's not
configured correctly and should change the netmask to /1.  The device
doesn't have any way to know that, and it must respond to the
discovery commands both before and after it's properly configured.

Was hoping that you could make such a change _only_ on the computer
that's receiving the data - that way it's only one change, the devices
don't need any tweaking. But if it can't be, it can't be.

I've reread the protocol documentation and noticed that the device has
to respond not only to broadcasts to 255.255.255.255 but also to
subnet broadcasts send to subnets it's not on.  That pretty much
clinches the requirement to use a raw socket. :/

Sounds to me like someone majorly abused IP to do weird things. Looks
like you're stuck doing the same weirdness, in whatever way you can
manage :| Sorry.

Chris Angelico

 

[Grant Edwards]

Was hoping that you could make such a change _only_ on the computer
that's receiving the data - that way it's only one change, the devices
don't need any tweaking. But if it can't be, it can't be.

There can by any number of devices that have to receive the broadcast
packet, and any of them can be on different subnets (or have no IP
address at all).

It turns out that some OSes (BSD and some flavors of Windows) can't
broadcast to 255.255.255.255, only to the subnet broadcast address.
Hence the requirement for the devices to be able to receive a subnet
broadcast regardless of their IP address.

Sounds to me like someone majorly abused IP to do weird things.

Indeed. The other option is to do something that's not based on IP
but done completely at the Ethernet layer. Implementing management
programs that can do that can be very nasty on Windows, which
unfortunately matters to most customers.

So you bite the bullet on the device end and implement all sorts of
weirdness in order to allow the management program to use standard
UDP.

Looks like you're stuck doing the same weirdness, in whatever way you
can manage

Yup. It doesn't even appear that it can be done with a raw UDP
socket. According to all of the documentation I can find, such a
socket is supposed to receive copies of all UDP packets seen by the
kernel, but it doesn't. Even if I use a raw UDP socket, the kernel is
still dropping broadcast packets whose source address don't match any
interface's subnet.

AFAICT, I'm going to have to go completely raw and process in
user-space every single IP packet recieved. That's going to be
brutal on the CPU...

 

[Dan Stromberg]

That would require that the device somehow knows that it's not
configured correctly and should change the netmask to /1.  The device
doesn't have any way to know that, and it must respond to the
discovery commands both before and after it's properly configured.

- Actually, you Might be able to configure your device to have a
netmask of 0.0.0.0, IP address of 255.255.255.255 and broadcast of
255.255.255.255.
- I've seen something a bit similar used for detecting IP address
conflicts automatically.
- A network guru I used to work with told me that you could configure
a machine with a broadcast of 255.255.255.255 more simply than messing
around with the netmask, while still achieving the same result for
general purpose networking.

I've reread the protocol documentation and noticed that the device has
to respond not only to broadcasts to 255.255.255.255 but also to
subnet broadcasts send to subnets it's not on.  That pretty much
clinches the requirement to use a raw socket. :/

With a netmask of 0.0.0.0, I suspect you will receive all broadcasts
on the wire, given appropriate listening code.

You could probably also modify a copy of tshark or tcpdump to flush
after every line of output (or run an unmodified one on a pty to avoid
maintaining your own copy of the binary), and parse that output in
Python. That should make the Python code pretty simple.

There's an old program called "pty" that makes it easy to run
something on a pty, to get unbuffered output - it's in
comp.sources.unix volumes 22, 23 and 25; it's written in C. You'd
just open a subprocess with no buffering on the python side, that runs
tcpdump or tshark under pty. Beware though - the pty program predates
GNU autoconf, so might be a little involved to compile.

I agree though that you're kind of pushing IP in a direction it wasn't
intended to go.

 

[Dan Stromberg]

I agree though that you're kind of pushing IP in a direction it wasn't
intended to go.

It just occurred to me: You might get some additional mileage out of
popping the network adapter into promiscuous mode. In fact, it Might
be necessary irrespective of the rest of your approach.

 

[Roy Smith]

I'm trying to implement a device discovery/configuration protocol that
uses UDP broadcast packets to discover specific types of devices on
the local Ethernet segment. The management program broadcasts a
discovery command to a particular UDP port. All devices who get that
packet are expected to answer regardless of thier current IP address.

Have you considered what will happen if you have, say, 1000 such
devices, and they all respond at the same time?

 

[Heiko Wundram]

Am 20.04.2011 01:54, schrieb Grant Edwards:

I guess the problem is that I expected to receive a packet on an
interface anytime a packet was received with a destination IP address
that matched that of the the interface. Apprently there's some
filtering in the network stack based on the _source_ address as well
(that seems very counter-intuitive to me).

Just to pitch in here (because nobody's mentioned it yet AFAICT): yes,
there's a filtering done (at least under Linux, and I'd guess something
similar on xBSD too) to packets based on the source address coming in on
an interface, and it's called the reverse path filter and is on by
default (the tunable on Linux is /proc/sys/net/ipv4/conf/*/rp_filter).

The idea behind the reverse path filter is that your machine won't
accept packets coming in over an interface when a return packet (i.e.,
the presumed response) won't be routed over the same interface, and from
what I gather, this is what makes the TCP/IP stack drop the packets
because your machine will not route packets to 192.168.x.x over the same
interface it sees the packet coming in. This is a _security_ feature,
because it makes address spoofing harder.

If you need to see the packets regardless, either use a promiscuous mode
sniffer (i.e., tcpdump, but that's relatively easy to mirror in Python
using SOCK_RAW, capturing packets at the ethernet level), or add a route
on your system for the 192.168.x.x network on the same interface.

HTH!

 

[Thomas Heller]

Am 20.04.2011 00:21, schrieb Grant Edwards:

I'm have problems figuring out how to receive UDP broadcast packets on
Linux. [...]

On the receiving machine, I've used tcpdump to verify that broadcast
packets are being seen and have a destination IP of 255.255.255.255 and
destination MAC of ff:ff:ff:ff:ff:ff [...]
But, the receiving Python program never sees any packets unless the
_source_ IP address in the packets is on the same subnet as the
receiving machine. In this test case, the receiving machine has an IP
address of 172.16.12.34/16. If I change the receiving machine's IP
address to 10.0.0.123, then the receiving program sees the packets.

Even though the destination address is 255.255.255.255, the receiving
machine appears to discard the packets based on the _source_ IP. Can
anybody provide example Python code for Linux that receives UDP
broadcast packets regardless of their source IP address?

This probably is more of a Linux networking question than a Python
question, but I'm hoping somebody has solved this problem in Python.

You must set the network interface to promiscous mode on the receiving side:

os.system("ifconfig eth0 promisc")

Thomas

 

[Adam Tauno Williams]

Have you looked at the source for Apple's Bonjour?

On LINUX this is called "avahi", which has Python bindings. Avahi
auto-configuration / discovery works very well.

<http://avahi.org/>
<http://freshmeat.net/projects/avahi>
<http://pypi.python.org/pypi/pybonjour/1.1.1>
<http://avahi.org/wiki/PythonBrowseExample>

See also:
<http://pypi.python.org/pypi/pybonjour/1.1.1>
at least you may be able to lift code from them (License is non-viral
MIT)