Problem with httplib and HEAD request

[Mitch.Garnaat]

Hi -

I'm writing some Python code to interact with Amazon's S3 service. One
feature of S3 is that it will allow you to use the HTTP HEAD request to
retrieve metadata about an S3 object without actually retrieving the
content of the object. In trying to use this feature, I'm running into
what I think is a bug in httplib.py.

If the resource I perform the HEAD request on exists within S3 then
everything is fine. If, however, the resource doesn't exist then a 404
is returned. If the request had been a GET rather than a HEAD the body
of the response would be some XML that gave further information about
the error status returned. However, for a HEAD request no body can be
returned so the body is empty. The error response, for either HEAD or
GET, does include the following header:

transfer-encoding: chunked

And this seems to be causing confusion in httplib.py. I have to call
the read method of the response or else I get a ResponseNotReady
exception on my next request. When I attempt to read the body of the
error response to the HEAD request it is passed onto the _read_chunked
method where it immediately tries to do a readline() on the connection
which, of course, hangs because no body has been sent in the response.

So, I know this is kind of a boundary condition and perhaps it hasn't
come up before but it seems that the behavior of httplib is incorrect.
I think something needs to be added to the _read_chunked method which
checks if the request is a GET and, if so, just reads zero bytes from
the socket and returns.

Any comments? Thanks,

Mitch

 

[Mitch.Garnaat]

For what it's worth, I made the following change to httplib.py and it
seems to have solved my problems:

c455
< if self.chunked:
---

if self.chunked and self._method != 'HEAD':

I'm using Python 2.4.1 on MacOS.

Mitch