urllib - changing the user agent

[Fuzzyman]

I'm writing a function that will query the comp.lang.python newsgroup
via google groups....... (I haven't got nntp access from work..)

I'm using urllib (for the first time)..... and google don't seem very
keen to let me search the group from within a program - the returned
pages all tell me 'you're not allowed to do that'

I read in the urllib manual pages :

class URLopener( [proxies[, **x509]])

Base class for opening and reading URLs. Unless you need to support
opening objects using schemes other than http:, ftp:, gopher: or
file:, you probably want to use FancyURLopener.
By default, the URLopener class sends a User-Agent: header of
"urllib/VVV", where VVV is the urllib version number. Applications can
define their own User-Agent: header by subclassing URLopener or
FancyURLopener and setting the instance attribute version to an
appropriate string value before the open() method is called.


Could anyone tell me how to subclass this correctly with the version
attribute set and what text string I should use to mimic Internet
explorer and/or mozilla ?


Ta

Fuzzy

 

[John J. Lee]

I'm using urllib (for the first time)..... and google don't seem very
keen to let me search the group from within a program - the returned
pages all tell me 'you're not allowed to do that'

Don't do it, then. Use the google API (mind you, not certain that
google groups is part of that).

[...]

Could anyone tell me how to subclass this correctly with the version
attribute set and what text string I should use to mimic Internet
explorer and/or mozilla ?

IIRC,

opener.addheaders = [("User-agent", "whatever")]

Use a program like ethereal to find what your browser sends for
"whatever".


John

 

[chitanom chantavong]

I'm writing a function that will query the comp.lang.python newsgroup
via google groups....... (I haven't got nntp access from work..)

I'm using urllib (for the first time)..... and google don't seem very
keen to let me search the group from within a program - the returned
pages all tell me 'you're not allowed to do that'

I read in the urllib manual pages :

class URLopener( [proxies[, **x509]])

Base class for opening and reading URLs. Unless you need to support
opening objects using schemes other than http:, ftp:, gopher: or
file:, you probably want to use FancyURLopener.
By default, the URLopener class sends a User-Agent: header of
"urllib/VVV", where VVV is the urllib version number. Applications can
define their own User-Agent: header by subclassing URLopener or
FancyURLopener and setting the instance attribute version to an
appropriate string value before the open() method is called.


Could anyone tell me how to subclass this correctly with the version
attribute set and what text string I should use to mimic Internet
explorer and/or mozilla ?


Ta

Fuzzy

I normally use something like this for crawling webpages.

import urllib
urllib.URLopener.version = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT
5.0; T312461)'
urllib.FancyURLopener.prompt_user_passwd = lambda self, host, realm: (None,
None)

Anthony McDonald

 

[Terry Carroll]

On 9 Jan 2004 05:09:41 -0800, (e-mail address removed) (Fuzzyman) wrote:

[ wants to change the user-agent in HTTP request from urllib ]

Fuzzy --

I take the easy way out, and use urllib2, instead:

url = "http//www.spam.com/eggs.html"
req_headers = {
'User-Agent': 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)',
}
req = urllib2.Request(url, None, req_headers)

 

[Terry Carroll]

Don't do it, then. Use the google API (mind you, not certain that
google groups is part of that).

Can Google APIs be used to access Google Groups? ...

No. The Google Web APIs service can only be used to search
Google's main index of 3 billion web pages.

http://www.google.com/apis/api_faq.html#ini2

 

[John J. Lee]

On 9 Jan 2004 05:09:41 -0800, (e-mail address removed) (Fuzzyman) wrote:

[ wants to change the user-agent in HTTP request from urllib ]

Fuzzy --

I take the easy way out, and use urllib2, instead: [...]
req = urllib2.Request(url, None, req_headers)

or again, you can set .addheaders on OpenerDirector (which will cause
those headers to be added to all requests).


John

 

[Samuel Walters]

|Thus Spake John J. Lee On the now historical date of Fri, 09 Jan 2004
20:16:54 +0000|

or again, you can set .addheaders on OpenerDirector (which will cause
those headers to be added to all requests).

This, however, does not stop the original User-agent header to be sent,
and google still filters out the request. Instead, it just causes a
second user-agent to be sent.

Here is the very bad code I used to solve this problem. There are better
ways, I assure you, but it should point you in the right direction. You
should probably do this with inheritance, but for my quick script, this is
what I did.

----
#Override the default OpenerDirector Class Init.
#OpenerDirector *insists* on adding a User-Agent
#That some websites don't like.
foo = OpenerDirector.__init__
def bar(self, Agent='None'):
foo(self)
self.addheaders = [('User-Agent',Agent)]

OpenerDirector.__init__ = bar

 

[John J. Lee]

|Thus Spake John J. Lee On the now historical date of Fri, 09 Jan 2004
20:16:54 +0000|

(just to clarify, I meant an OpenerDirector instance, not the class
itself)

This, however, does not stop the original User-agent header to be sent,
and google still filters out the request. Instead, it just causes a
second user-agent to be sent.

No, it does stop them being sent. Perhaps you mutated the base class
..addheaders by mistake (using .append(("User-agent", "blah")), for
example)? Don't do that! Mutating class attributes is a bad idea.

Here is the very bad code I used to solve this problem. There are better
ways, I assure you, but it should point you in the right direction. You

[...]

No need to assure me of *that* <wink>. You can call a base class
constructor directly, you know. And clobbering the base class'
constructor is another very bad idea.


John

 

[Fuzzyman]

Thanks to all who posted help.
I think theres more than enough to get me through this - when I get a
chance I'll sit and have a play. I'm sure I'll have more
questions.......

Thanks

Fuzzy

|Thus Spake John J. Lee On the now historical date of Fri, 09 Jan 2004
20:16:54 +0000|

(just to clarify, I meant an OpenerDirector instance, not the class
itself)

This, however, does not stop the original User-agent header to be sent,
and google still filters out the request. Instead, it just causes a
second user-agent to be sent.

No, it does stop them being sent. Perhaps you mutated the base class
.addheaders by mistake (using .append(("User-agent", "blah")), for
example)? Don't do that! Mutating class attributes is a bad idea.

Here is the very bad code I used to solve this problem. There are better
ways, I assure you, but it should point you in the right direction. You

[...]

No need to assure me of *that* <wink>. You can call a base class
constructor directly, you know. And clobbering the base class'
constructor is another very bad idea.


John