Re: filedescriptor out of range in select()

[Andrew Bennetts]

Spambayes CVS, Python 2.2.3, FreeBSD 5.1

On my machine I'm lunching the pop3proxy with cron at startup, and all
is fine.

But... sometimes during startup, or after a one or two days, the process
exits with the following error...

Is a BSD select related problem? Is an asyncore problem? Is a spambayes
problem?

I'm not very familiar with BSD, asyncore or spambayes, but I think
"filedescriptor out of range in select()" usually means that the process is
trying to handle too many connections at once -- maybe the sockets aren't
getting closed, or something.

You might like to check the output of netstat when it happens, too see if
there's an outrageously large number of connections to your pop3proxy. I'm
just guessing, though.

-Andrew.

 

[Erik Max Francis]

I'm not very familiar with BSD, asyncore or spambayes, but I think
"filedescriptor out of range in select()" usually means that the
process is
trying to handle too many connections at once -- maybe the sockets
aren't
getting closed, or something.

I strongly suspect it means that the file descriptor (or the result of
calling the .fileno method of a socket-like object) passed in one of the
objects is way out of range for being a valid file descriptor number.
If it were simply bad (i.e., not corresponding to any valid file within
that process but a legal value), the error would be "Bad file
descriptor":

max@oxygen:~% python
Python 2.2.3 (#1, May 31 2003, 21:31:33)
[GCC egcs-2.91.66 19990314/Linux (egcs-1.1.2 release)] on linux2
Type "help", "copyright", "credits" or "license" for more information.

import select
select.select([4], [], [], 0)

Traceback (most recent call last):

select.select([10000000], [], [], 0)

Traceback (most recent call last):
File "<stdin>", line 1, in ?
ValueError: filedescriptor out of range in select()

 

[Paolo Invernizzi]

but that doesn't necessarily mean that the #define in an include file
somewhere automagically picks up the kernel configuration, does it?

You are right, obviously.

(a quick googling brings up some posts about FreeBSD machines with
FD_SETSIZE=256 and maxuserfiles>1024 etc. dunno if they've made
their configuration tools smarter, but in any case, "poll" is the "right
way" to deal with large number of file handles (compared to "select",
at least -- there are better but platform-specific ways to do things
if you want to...))

Right again.

Many thanks for the help

 

[Paolo Invernizzi]

but I'm pretty sure asyncore supports the "poll" interface. try changing
the asyncore.loop call in Dibbler.py to:

asyncore.loop(map=context._map, use_poll=1)

and see what happens.

Here we are again....
Too many open files... with poll too...

---
Paolo

SpamBayes POP3 Proxy Beta1, version 0.1 (May 2003),
using SpamBayes POP3 Proxy Web Interface Alpha2, version 0.02
and engine SpamBayes Beta1, version 0.1 (May 2003).

Loading database... Done.
SMTP Listener on port 2525 is proxying 127.0.0.1:25
Listener on port 8110 is proxying kpnqwest.it:110
User interface url is http://localhost:8880/
error: uncaptured python exception, closing channel
<__main__.BayesProxyListener listening :8110 at 0x83c9f6c>
(socket.error24, 'Too many open files')
[/usr/local/lib/python2.2/asyncore.py|poll3|184]
[/usr/local/lib/python2.2/asyncore.py|handle_read_event|391]
[/usr/local/lib/python2.2/site-packages/spambayes/Dibbler.py|handle_accept|280]
[/usr/local/bin/pop3proxy.py|__init__|337]
[/usr/local/bin/pop3proxy.py|__init__|180]
[/usr/local/bin/pop3proxy.py|__init__|134]
[/usr/local/lib/python2.2/asyncore.py|create_socket|260])
error: uncaptured python exception, closing channel <__main__.BayesProxy
connected 192.168.0.2:63485 at 0x84d8aac>
(exceptions.AttributeError:'_socket.socket' object has no attribute
'isClosed' [/usr/local/lib/python2.2/asyncore.py|poll3|184]
[/usr/local/lib/python2.2/asyncore.py|handle_read_event|397]
[/usr/local/lib/python2.2/asynchat.py|handle_read|82]
[/usr/local/bin/pop3proxy.py|recv|360]
[/usr/local/lib/python2.2/asyncore.py|recv|353]
[/usr/local/lib/python2.2/asynchat.py|handle_close|149]
[/usr/local/bin/pop3proxy.py|close|368]
[/usr/local/lib/python2.2/asyncore.py|__getattr__|372])
Traceback (most recent call last):
File "/usr/local/bin/pop3proxy.py", line 731, in ?
run()
File "/usr/local/bin/pop3proxy.py", line 725, in run
main(state.servers, state.proxyPorts, state.uiPort, state.launchUI)
File "/usr/local/bin/pop3proxy.py", line 664, in main
Dibbler.run(launchBrowser=launchUI)
File "/usr/local/lib/python2.2/site-packages/spambayes/Dibbler.py",
line 528, in run
asyncore.loop(map=context._map,use_poll=1)
File "/usr/local/lib/python2.2/asyncore.py", line 206, in loop
poll_fun (timeout, map)
File "/usr/local/lib/python2.2/asyncore.py", line 190, in poll3
obj.handle_error()
File "/usr/local/lib/python2.2/site-packages/spambayes/Dibbler.py",
line 205, in handle_error
asynchat.async_chat.handle_error(self)
File "/usr/local/lib/python2.2/asyncore.py", line 427, in handle_error
self.close()
File "/usr/local/bin/pop3proxy.py", line 368, in close
if not self.isClosed:
File "/usr/local/lib/python2.2/asyncore.py", line 372, in __getattr__
return getattr (self.socket, attr)
AttributeError: '_socket.socket' object has no attribute 'isClosed'

 

[Steve Holden]

but I'm pretty sure asyncore supports the "poll" interface. try changing
the asyncore.loop call in Dibbler.py to:

asyncore.loop(map=context._map, use_poll=1)

and see what happens.

Here we are again....
Too many open files... with poll too...

---
Paolo

SpamBayes POP3 Proxy Beta1, version 0.1 (May 2003),
using SpamBayes POP3 Proxy Web Interface Alpha2, version 0.02
and engine SpamBayes Beta1, version 0.1 (May 2003).

Loading database... Done.
SMTP Listener on port 2525 is proxying 127.0.0.1:25
Listener on port 8110 is proxying kpnqwest.it:110
User interface url is http://localhost:8880/
error: uncaptured python exception, closing channel
<__main__.BayesProxyListener listening :8110 at 0x83c9f6c>
(socket.error24, 'Too many open files') [...]
Traceback (most recent call last):
File "/usr/local/bin/pop3proxy.py", line 731, in ?
run()
File "/usr/local/bin/pop3proxy.py", line 725, in run
main(state.servers, state.proxyPorts, state.uiPort, state.launchUI)
File "/usr/local/bin/pop3proxy.py", line 664, in main
Dibbler.run(launchBrowser=launchUI)
File "/usr/local/lib/python2.2/site-packages/spambayes/Dibbler.py",
line 528, in run
asyncore.loop(map=context._map,use_poll=1)
File "/usr/local/lib/python2.2/asyncore.py", line 206, in loop
poll_fun (timeout, map)
File "/usr/local/lib/python2.2/asyncore.py", line 190, in poll3
obj.handle_error()
File "/usr/local/lib/python2.2/site-packages/spambayes/Dibbler.py",
line 205, in handle_error
asynchat.async_chat.handle_error(self)
File "/usr/local/lib/python2.2/asyncore.py", line 427, in handle_error
self.close()
File "/usr/local/bin/pop3proxy.py", line 368, in close
if not self.isClosed:
File "/usr/local/lib/python2.2/asyncore.py", line 372, in __getattr__
return getattr (self.socket, attr)
AttributeError: '_socket.socket' object has no attribute 'isClosed'

I suspect that the "too many open files" is simply a symptom of a deeper
problem, in that your sockets aren't being correctly closed due to a
programming error. The asyncore framework will happily continue to run even
if errors occur in the event-handling routines, cneatly demonstrating the
dangers of an unlimited "try:/except:".

There's a clear implication here that a BayesProxy instance isn't running
its __init__() method, because if it *were* running it the isClosed
attribute would be defined.

Are you perchance subclassing Bayesproxy and overriding its __init__()
rather than extending it? If so then try adding

BayesProxy.__init__(self, clientSocket, serverName, serverPort)

or equivalent to your subclass' __init__().

regards

 

[Paolo Invernizzi]

I suspect that the "too many open files" is simply a symptom of a deeper
problem, in that your sockets aren't being correctly closed due to a
programming error. The asyncore framework will happily continue to run even
if errors occur in the event-handling routines, cneatly demonstrating the
dangers of an unlimited "try:/except:".

There's a clear implication here that a BayesProxy instance isn't running
its __init__() method, because if it *were* running it the isClosed
attribute would be defined.

I'll check this today...

Are you perchance subclassing Bayesproxy and overriding its __init__()
rather than extending it?

Nono, I'm using the 'plain' pop3proxy.py of spambayes... I'm not
extending the framework...