sending a file chunk by chunk instead as a whole to a web server

[Sanjeeb]

Hi,
I have a web client which send a file to a server as multipart form
data, the sending of data is from
http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.

I dont want to open the whole file to memory(at cliend end) and then
send, i just want to send part by part, say chunk of 1024 bytes to the
server and then assemble at the server end.

Could some one suggest what would be the best way to do this?

Regards
-Sanjeeb

 

[Sanjeeb]

Hi,
I have a web client which send a file to a server as multipart form
data, the sending of data is fromhttp://code.activestate.com/recipes/146306-http-client-to-post-using-....

I dont want to open the whole file to memory(at cliend end) and then
send, i just want to send part by part, say chunk of 1024 bytes to the
server and then assemble at the server end.

Could some one suggest what would be the best way to do this?

Regards
-Sanjeeb

Hey any taker for this question???

 

[Kushal Kumaran]

Hi,
I have a web client which send a file to a server as multipart form
data, the sending of data is from
http://code.activestate.com/recipes/146306-http-client-to-post-using-multipartform-data/.

I dont want to open the whole file to memory(at cliend end) and then
send, i just want to send part by part, say chunk of 1024 bytes to the
server and then assemble at the server end.

Could some one suggest what would be the best way to do this?

There's no reason why sending the whole file implies reading the whole
file into memory at one time. You can just read your desired chunk
size and send it, then read the next chunk, and so on. You might have
to first find the total size to calculate what to set Content-Length
to.

ISTR questions of this nature have been asked in the past on this list.

 

[Ryan Kelly]

There's no reason why sending the whole file implies reading the whole
file into memory at one time. You can just read your desired chunk
size and send it, then read the next chunk, and so on. You might have
to first find the total size to calculate what to set Content-Length
to.

More concretely, you would restructure the encode_multipart_formdata()
function as a generator, yielding chunks of data to send one at a time.
Something like this:

def post_multipart(host,selector,fields,files):
...snip...
h.endheaders()
for chunk in encode_multipart_formdata_chunks(fields,files):
h.send(chunk)
errcode, errmsg, headers = h.getreply()
return h.file.read()


def encode_multipart_formdata(fields,files):
BOUNDARY = '----------ThIs_Is_tHe_bouNdaRY_$'
CRLF = '\r\n'
for (key, value) in fields:
yield '--' + BOUNDARY
yield 'Content-Disposition: form-data; name="%s"' % key
yield ''
yield value
for (key, filename, value) in files:
yield '--' + BOUNDARY
yield 'Content-Disposition: form-data; name="%s"; filename="%s"' % (key, filename)
...etc...
...etc...

There are many improvements to make, but this should get you started.
For example, you'll need to calculate the total content-length rather
than just calling len(body) to obtain it. That's left as an exercise to
the reader.


Ryan


--
Ryan Kelly
http://www.rfk.id.au | This message is digitally signed. Please visit
(e-mail address removed) | http://www.rfk.id.au/ramblings/gpg/ for details


-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)

iEYEABECAAYFAkxXrpwACgkQfI5S64uP50q+fACgzcvVbFuG25WI1B/9Daqk+Zvp
LVIAoMbwFNlvrPsz5vZqwjR8s8P5FHhY
=rhPT
-----END PGP SIGNATURE-----