No way to set a timeout in "urllib".

[John Nagle]

There's no way to set a timeout if you use "urllib" to open a URL.
"HTTP", which "urllib" uses, supports this, but the functionality
is lost at the "urllib" level.

It's not available via "class URLopener" or "FancyURLopener", either.

There is a non-thread-safe workaround from 2003 at

http://mail.python.org/pipermail/python-bugs-list/2003-September/020405.html

but it was rejected as a feature at

https://sourceforge.net/tracker/?func=detail&atid=105470&aid=803634&group_id=5470

without anything better going in. Despite this, current documentation
recommends that approach:

http://svn.python.org/projects/python/trunk/Doc/howto/urllib2.rst

Someone proposed to fix this

http://mail.python.org/pipermail/python-dev/2006-July/066967.html

but was discouraged from doing so.

The code was forked by Zope as a workaround in 2003:

http://pywebsvcs.sourceforge.net/apidocs/wstools/Utility.html

but that's not in the mainstream Python tree.

The correct fix would probably be to add methods to class
URLopener to control this; that's the usual way of handling special
URL opening situations.

John Nagle

 

[skip]

John> There's no way to set a timeout if you use "urllib" to open a URL.
John> "HTTP", which "urllib" uses, supports this, but the functionality
John> is lost at the "urllib" level.

John> It's not available via "class URLopener" or "FancyURLopener",
John> either.

John> There is a non-thread-safe workaround from 2003 at

...

This topic has come up several times since timeouts were added to socket.
Each time we've asked for a patch that adds timeouts in a rational manner to
all the stuff layered on top of the socket module (httplib, ftplib, etc). As
far as I know it's apparently never been important enough for anyone to rise
to the challenge. If I remember next spring perhaps I'll submit it as a
possible Google Summer of Code proposal.

John> The correct fix would probably be to add methods to class
John> URLopener to control this; that's the usual way of handling
John> special URL opening situations.

The correct way would be to deal with it at the httplib level, then
percolate it up to urllib2. Urllib should probably not be extended any
further.

Skip

 

[John Nagle]

John> There's no way to set a timeout if you use "urllib" to open a URL.
John> "HTTP", which "urllib" uses, supports this, but the functionality
John> is lost at the "urllib" level.

John> It's not available via "class URLopener" or "FancyURLopener",
John> either.

John> There is a non-thread-safe workaround from 2003 at

...

This topic has come up several times since timeouts were added to socket.
Each time we've asked for a patch that adds timeouts in a rational manner to
all the stuff layered on top of the socket module (httplib, ftplib, etc). As
far as I know it's apparently never been important enough for anyone to rise
to the challenge. If I remember next spring perhaps I'll submit it as a
possible Google Summer of Code proposal.

It ought not to take more than a day or two, although it is annoying
that you have to deal with the problem at several levels.

If you're looking for a Summer of Code project, consider finishing
the Python SSL library, which doesn't even check certificates, let alone
revocation lists. It's a pure interface problem. OpenSSL does all the
hard parts, but the Python glue code is crude and incomplete.

John Nagle

 

[skip]

John> If you're looking for a Summer of Code project, ...

I'm not. I'm about 25 years out of grad school. ;-)

Skip

 

[John J. Lee]

There's no way to set a timeout if you use "urllib" to open a URL.
"HTTP", which "urllib" uses, supports this, but the functionality
is lost at the "urllib" level.

It's not available via "class URLopener" or "FancyURLopener", either.

There is a non-thread-safe workaround from 2003 at

http://mail.python.org/pipermail/python-bugs-list/2003-September/020405.html

but it was rejected as a feature at

https://sourceforge.net/tracker/?func=detail&atid=105470&aid=803634&group_id=5470

without anything better going in. Despite this, current documentation
recommends that approach:

http://svn.python.org/projects/python/trunk/Doc/howto/urllib2.rst

And...? What specifically are you complaining about?

Just as a matter of fact (I'm not grumpy about it): I see from your
message that you already know that there is not a shortage of people
who spot the lack of this kind of feature. The shortage is of people
who will actually do the job of adding the feature -- most
importantly, people who will contribute high quality patches,
including tests and docs, and people who will review other people's
patches.

Someone proposed to fix this

http://mail.python.org/pipermail/python-dev/2006-July/066967.html

but was discouraged from doing so.

[...]

While it might be discouraging to be confronted with tiresome
realities of schedules and resources, those are the realities.

The flip side is that you can make a real difference by putting in
some work.

Looking forward to your patch


John