[Newbie] Is there any method to urlretrieve to a file the html source

[Aldo Ceccarelli]

Hello All,
I apologize for posting a question on this but I could not find a
complete answer after reading and searching so far

My problem is that I'd like to store the html source of a certain web
url but when I try via urllib / urllib2 reads or urlretrieve I get
only a part of the contents (ist type is text/html).

Metainformation associated with the url show that it has a:

Transfer-Encoding: chunked

and this seems the reason why I can get only a part of it.

Can anybody address me to a possible solution?

Many thanks in advance!
WKR Aldo

 

[Thinker]

Hello All,
I apologize for posting a question on this but I could not find a
complete answer after reading and searching so far

My problem is that I'd like to store the html source of a certain web
url but when I try via urllib / urllib2 reads or urlretrieve I get
only a part of the contents (ist type is text/html).

Metainformation associated with the url show that it has a:

Transfer-Encoding: chunked

and this seems the reason why I can get only a part of it.

Please show your code, here. urllib doesn't always return full content
in one time. I guess it is the problem.

 

[Aldo Ceccarelli]

Thank you All for your suggestions:

I could finally assess that

besides partial read from chunked url - by-passed thanks to a socket
approach:

f.i. this was good for me: http://python.about.com/od/networkingwithpython/ss/beg_web_client_9.htm

I have still two kind of problems that quit this post's aims (original
url is redirected and has password protections too, I will go for
these now)

Again SUPER thanks to You All Aldo

 

[7stud]

Thank you All for your suggestions:

I could finally assess that

besides partial read from chunked url - by-passed thanks to a socket
approach:

f.i. this was good for me:http://python.about.com/od/networkingwithpython/ss/beg_web_client_9.htm

I have still two kind of problems that quit this post's aims (original
url is redirected and has password protections too, I will go for
these now)

Again SUPER thanks to You All Aldo

Don't use that code--it's merely a bare bones example. Post the url
that is giving you problems.