socket.error 24: too many open files

[TheDavidFactor]

I'm new to python, but have been writing programs in other languages
for about 15 years now. As part of my job I develop applications that
interface with Asterisk in various ways. As a way of getting my feet
wet I decided to try to rewrite an outbound call script that I have in
another language into python. It's a deamon that runs on a linux box
and every 15 seconds it checks a MySQL table for new records, if there
are any it creates a .call file on the Asterisk server using ssh, it
also checks the Asterisk server, again via ssh, for any finished calls
and if there are any it reads the .call file and writes the result in
to the MySQL table.

I did a google search for ssh and a python script that someone had
written that wraps up some paramiko calls to make the ssh interaction
easier. I'm also using MySQLdb for the MySQL access. It sleeps for 15
seconds then creates an ssh connection the asterisk server, checks the
MySQL table, checks the asterisk server, closes the connection and
goes back to sleep. After it has run for an hour or so I get the error
in the subject when the ssh class tries to create a new socket. I have
double checked that it is closing the socket. I don't know what else
to check, any suggestions would be much appreciated.

 

[Roy Smith]

I have double checked that it is closing the socket. I don't know what
else to check, any suggestions would be much appreciated.

All of the symptoms you report point to sockets not getting closed. What
does "double checked" mean? Don't rely on __del__() closing sockets,
because __del__() may never get called.

Here's a few ways to look for sockets that aren't getting closed:

1) Run netstat.

2) Look in /proc.

3) This is the cute one; iterate over all possible descriptors from 0 to
`ulimit -n`. For each one, dup() it, then close the dupe. If dup() didn't
raise an exception, the descriptor was valid. Count the ones that don't
fail, and you know how many descriptors are open at any time. This is
useful for tracking descriptor leaks (which sounds like what you've got).

 

[Bryan Olson]

[...] It's a deamon that runs on a linux box
and every 15 seconds it checks a MySQL table for new records, if there
are any it creates a .call file on the Asterisk server using ssh, it
also checks the Asterisk server, again via ssh, for any finished calls
and if there are any it reads the .call file and writes the result in
to the MySQL table.

[...] After it has run for an hour or so I get the error
in the subject when the ssh class tries to create a new socket. I have
double checked that it is closing the socket. I don't know what else
to check, any suggestions would be much appreciated.

Is it possible you keep accumulating MySQLdb connection or cursor
objects and don't close() them? (I don't know the innards of MySQLdb,
but it's something to check.)

One thing you might try is to regularly log the filno() of your sockets.