urllib accept-language doesn't have any effect

[Martin Bachwerk]

Hello,

I'm trying to load a couple of pages using the urllib2 module. The
problem is that I live in Germany and some sites seem to look at the IP
of the client and forward him to a localized page.. Here's an example of
the code, how I want to access google.com main english page, but get
German instead. (For those of you who live in US, you will probably get
correct results.. try emulating with 'fr' in accepted languages or
something)

opener = urllib2.build_opener()
opener.addheaders = [('Host', 'www.google.com'),
('Accept-Language','en-gb,en;q=0.5'), ('User-agent', 'Mozilla/5.0
(Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.1) Gecko/2008070208
Firefox/3.0.1')]
webfile = opener.open(url)

Any help would be greatly appreciated!

Thank you!

Martin

 

[Diez B. Roggisch]

Hello,

I'm trying to load a couple of pages using the urllib2 module. The
problem is that I live in Germany and some sites seem to look at the IP
of the client and forward him to a localized page.. Here's an example of
the code, how I want to access google.com main english page, but get
German instead. (For those of you who live in US, you will probably get
correct results.. try emulating with 'fr' in accepted languages or
something)

opener = urllib2.build_opener()
opener.addheaders = [('Host', 'www.google.com'),
('Accept-Language','en-gb,en;q=0.5'), ('User-agent', 'Mozilla/5.0
(Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.1) Gecko/2008070208
Firefox/3.0.1')]
webfile = opener.open(url)

Any help would be greatly appreciated!

What is your actual problem - does it work with *other* requests you somehow
produce (browser or some such)?

In the end, it's up to the site you query to return you whatever they see
fit. Nothing urllib or python in general can change.

Diez

 

[Martin Bachwerk]

Hi,

yes, well my browser settings are set to display sites in following
languages "en-gb" then "en".

As a matter of fact, Google does indeed show me the German site first,
before I click on "go to google.com" and it probably stores a cookie to
remember that.

But a site like gizmodo.com forwards me directly to the English site
(even though they have a German version available). But the Python call
returns the German version. How does that make sense?

Cheers,
Martin

Martin Bachwerk wrote:

Hello,

I'm trying to load a couple of pages using the urllib2 module. The
problem is that I live in Germany and some sites seem to look at the IP
of the client and forward him to a localized page.. Here's an example of
the code, how I want to access google.com main english page, but get
German instead. (For those of you who live in US, you will probably get
correct results.. try emulating with 'fr' in accepted languages or
something)

opener = urllib2.build_opener()
opener.addheaders = [('Host', 'www.google.com'),
('Accept-Language','en-gb,en;q=0.5'), ('User-agent', 'Mozilla/5.0
(Windows; U; Windows NT 5.1; en-GB; rv:1.9.0.1) Gecko/2008070208
Firefox/3.0.1')]
webfile = opener.open(url)

Any help would be greatly appreciated!

What is your actual problem - does it work with *other* requests you somehow
produce (browser or some such)?

In the end, it's up to the site you query to return you whatever they see
fit. Nothing urllib or python in general can change.

Diez

 

[Diez B. Roggisch]

Hi,

yes, well my browser settings are set to display sites in following
languages "en-gb" then "en".

As a matter of fact, Google does indeed show me the German site first,
before I click on "go to google.com" and it probably stores a cookie to
remember that.

But a site like gizmodo.com forwards me directly to the English site
(even though they have a German version available). But the Python call
returns the German version. How does that make sense?

I don't know. You need to have a look at what the browser communicates,
through a tool like tcpmon or wireshark. Once you've gathered all needed
header-information, you can mimic the same request using urllib.

Diez