gethostbyname blocking

[marc wyburn]

Hi, I am writing an asynchronous ping app to check if 1000s of hosts
are alive very quickly. Everything works extremely quickly unless the
host name doesn't have a DNS record.

when calling socket.gethostbyname if there is no record for the host
the result seems to block all other threads. As an example I have 6
threads running and if I pass the class below a Queue with about 30
valid addresses with one invalid address in the middle the thread that
the exception occurs in seems to block the others.

I'm pretty new to threading so am not sure if my threading code is
wrong or whether I need to look at how gethostbyname? When there list
is made entirely of valid host names I get the responses back very
very quickly.

class HostThread(threading.Thread):
def __init__(self, Host_Queue):
threading.Thread.__init__(self)
self.Host_Queue = Host_Queue
def run(self):
while True:
# Create a new IP packet and set its source and
destination addresses.
host = self.Host_Queue.get()
try:
dest_addr, alias, ipaddrlist =
socket.gethostbyname_ex(host)
except socket.gaierror, (errorno, details):
print '%s: %s' %(host,details)
else:
print '%s: %s' %(host, ipaddrlist)
self.Host_Queue.task_done()

Thanks, Marc.

 

[Piet van Oostrum]

MW> Hi, I am writing an asynchronous ping app to check if 1000s of hosts
MW> are alive very quickly. Everything works extremely quickly unless the
MW> host name doesn't have a DNS record.

MW> when calling socket.gethostbyname if there is no record for the host
MW> the result seems to block all other threads. As an example I have 6
MW> threads running and if I pass the class below a Queue with about 30
MW> valid addresses with one invalid address in the middle the thread that
MW> the exception occurs in seems to block the others.

What you could do is have two Queues, one with host names, and one with
results from gethostbyname. And an additional thread which gets from the
first queue, calls gethostbyname and puts the results in the second queue.
The ping threads then should get from the second queue. The lookup
thread could even do caching of the results if you often have to repeat
the pings for thew same host.

 

[marc wyburn]

What you could do is have two Queues, one with host names, and one with
results fromgethostbyname. And an additional thread which gets from the
first queue, callsgethostbynameand puts the results in the second queue.
The ping threads then should get from the second queue. The lookup
thread could even do caching of the results if you often have to repeat
the pings for thew same host.

Hi, I did try this but the socket blocked for so long that this was a
real bottleneck. It also seemed that whilst the DNS lookup thread was
blocking the other threads were also blocked. I guess this may have
something to do with the GIL but I'm still fairly new to threading. I
ended up using DNSPython which flys along. I have a thread creating
the ping packets and pinging them and another performing a select and
logging + filtering any ICMP replies. I may put the DNS code into
another thread but the whole script runs faster than I need it to as
it is, I've got time.sleeps in the ping thread as I'm scared of
saturating the local LAN during work hours.