Urllib2 urlopen and read - difference

[koranthala]

Hi,
Suppose I am doing the following:
req = urllib2.urlopen('http://www.python.org')
data = req.read()

When is the actual data received? is it done by the first line? or
is it done only when req.read() is used?
My understanding is that when urlopen is done itself, we would have
received all the data, and req.read() just reads it from the file
descriptor.
But, when I read the source code of pylot, it mentioned the
following:
resp = opener.open(request) # this sends the HTTP request
and returns as soon as it is done connecting and sending
connect_end_time = self.default_timer()
content = resp.read()
req_end_time = self.default_timer()

Here, it seems to suggest that the data is received only after you do
resp.read(), which made me all confused.

If someone could help me out, it would be much helpful.

 

[J. Cliff Dyer]

Hi,
Suppose I am doing the following:
req = urllib2.urlopen('http://www.python.org')
data = req.read()

When is the actual data received? is it done by the first line? or
is it done only when req.read() is used?
My understanding is that when urlopen is done itself, we would have
received all the data, and req.read() just reads it from the file
descriptor.
But, when I read the source code of pylot, it mentioned the
following:
resp = opener.open(request) # this sends the HTTP request
and returns as soon as it is done connecting and sending
connect_end_time = self.default_timer()
content = resp.read()
req_end_time = self.default_timer()

Here, it seems to suggest that the data is received only after you do
resp.read(), which made me all confused.

If someone could help me out, it would be much helpful.

My understanding (please correct me if I'm wrong), is that when you call
open, you send a request to the server, and get a response object back.
The server immediately begins sending data (you can't control when they
send it, once you've requested it). When you call read() on your
response object, it reads all the data it has already received, and if
that amount of data isn't sufficient to handle your read call, it blocks
until it has enough.

So your opener returns as soon as the request is sent, and read() blocks
if it doesn't have enough data to handle your request.

Cheers,
Cliff

 

[J. Cliff Dyer]

Hi,
Suppose I am doing the following:
req = urllib2.urlopen('http://www.python.org')
data = req.read()

When is the actual data received? is it done by the first line? or
is it done only when req.read() is used?
My understanding is that when urlopen is done itself, we would have
received all the data, and req.read() just reads it from the file
descriptor.
But, when I read the source code of pylot, it mentioned the
following:
resp = opener.open(request) # this sends the HTTP request
and returns as soon as it is done connecting and sending
connect_end_time = self.default_timer()
content = resp.read()
req_end_time = self.default_timer()

Here, it seems to suggest that the data is received only after you do
resp.read(), which made me all confused.

If someone could help me out, it would be much helpful.

My understanding (please correct me if I'm wrong), is that when you call
open, you send a request to the server, and get a response object back.
The server immediately begins sending data (you can't control when they
send it, once you've requested it). When you call read() on your
response object, it reads all the data it has already received, and if
that amount of data isn't sufficient to handle your read call, it blocks
until it has enough.

So your opener returns as soon as the request is sent, and read() blocks
if it doesn't have enough data to handle your request.

Cheers,
Cliff

 

[Aahz]

My understanding (please correct me if I'm wrong), is that when you call
open, you send a request to the server, and get a response object back.
The server immediately begins sending data (you can't control when they
send it, once you've requested it). When you call read() on your
response object, it reads all the data it has already received, and if
that amount of data isn't sufficient to handle your read call, it blocks
until it has enough.

So your opener returns as soon as the request is sent, and read() blocks
if it doesn't have enough data to handle your request.

Close. urlopen() returns after it receives the HTTP header (that's why
you can get an HTTP exception on e.g. 404 without the read()).