Making a persistent HTTP connection

[David Rasmussen]

I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?

/David

 

[Diez B. Roggisch]

I use urllib2 to do some simple HTTP communication with a web server. In
one "session", I do maybe 10-15 requests. It seems that urllib2 opens op
a connection every time I do a request. Can I somehow make it use _one_
persistent connection where I can do multiple GET->"receive data" passes
before the connection is closed?

Are you sure HTTP supports that? This would be news to me - which
doesn't mean much

And even if it works - what is the problem with connections being created?

Regards,

Diez

 

[Benjamin Niemann]

Are you sure HTTP supports that? This would be news to me - which
doesn't mean much

It does (HTTP/1.1 at least) and it's called 'keep-alive'.

And even if it works - what is the problem with connections being created?

Performance, network load...

 

[Alan Kennedy]

[David Rasmussen]
[Diez B. Roggisch]

Are you sure HTTP supports that?

Yes, HTTP 1.1 definitely supports multiple requests on the same connection.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1

Some HTTP 1.0 clients supported persistent connections through the use
of the non-standard "keep-alive" header.

And even if it works - what is the problem with connections being created?

The URL above describes the benefits of persistent connections. The
primary problem of the old style of one-request-per-connection is the
creation of more sockets than are necessary.

To the OP: neither urllib nor urllib2 implements persistent connections,
but httplib does. See the httplib documentation page for an example.

http://www.python.org/doc/2.4.2/lib/httplib-examples.html

However, even httplib is "synchronous", in that it cannot pipeline
requests: the response to the first request must be competely read
before a second request can be issued.

HTH,

 

[Alan Kennedy]

[David Rasmussen]
[Diez B. Roggisch]

Are you sure HTTP supports that?

Yes, HTTP 1.1 definitely supports multiple requests on the same
connection.

http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1

Some HTTP 1.0 clients supported persistent connections through the use
of the non-standard "keep-alive" header.

And even if it works - what is the problem with connections being created?

The URL above describes the benefits of persistent connections. The
primary problem of the old style of one-request-per-connection is the
creation of more sockets than are necessary.

To the OP: neither urllib nor urllib2 implements persistent
connections, but httplib does. See the httplib documentation page for
an example.

http://www.python.org/doc/2.4.2/lib/httplib-examples.html

However, even httplib is "synchronous", in that it cannot pipeline
requests: the response to the first request must be competely read
before a second request can be issued.

HTH,

 

[Piet van Oostrum]

AK> http://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1

AK> Some HTTP 1.0 clients supported persistent connections through the use
AK> of the non-standard "keep-alive" header.

AK> The URL above describes the benefits of persistent connections. The
AK> primary problem of the old style of one-request-per-connection is the
AK> creation of more sockets than are necessary.

Maybe even more important (and just briefly mentioned in the section
referred to above) is the latency introduced by the TCP setup and the slow
startup phase of TCP's congestion control. This calculation is one of the
exercises the students have to make in my networks class.