memory error with zipfile module

[Hari Sekhon]

I do

import zipfile
zip=zipfile.ZipFile('d:\somepath\cdimage.zip')
zip.namelist()
['someimage.iso']

then either of the two:

A) file('someimage.iso','w').write(zip.read('someimage.iso'))
or
B) content=zip.read('someimage.iso')

but both result in the same error:

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "D:\u\Python24\lib\zipfile.py", line 357, in read
bytes = dc.decompress(bytes)
MemoryError

I thought python was supposed to handle memory for you?

The python zipfile module is obviously broken...


Any advise?

 

[bruno at modulix]

I do

import zipfile
zip=zipfile.ZipFile('d:\somepath\cdimage.zip')
zip.namelist()
['someimage.iso']

then either of the two:

A) file('someimage.iso','w').write(zip.read('someimage.iso'))
or
B) content=zip.read('someimage.iso')

but both result in the same error:

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "D:\u\Python24\lib\zipfile.py", line 357, in read
bytes = dc.decompress(bytes)
MemoryError

I thought python was supposed to handle memory for you?

Err... This doesn't mean that it bypasses system's memory management.

http://pyref.infogami.com/MemoryError

http://mail.zope.org/pipermail/zope/2004-October/153882.html
"""
MemoryError is raised by Python when an underlying (OS-level) allocation
fails.
(...)
Normally this would mean that you were out of even virtual memory
(swap), but it could also be a symptom of a libc bug, a bad RAM chip, etc.
"""

What do you think will append if you try to allocate a huge block when
you've already eaten all available memory ? Do you really hope that
Python will give you extra ram for free ?-)

Please try this code:

import zipfile
zip=zipfile.ZipFile('d:\somepath\cdimage.zip')
info = zip.getinfo('someimage.iso')
csize = info.compress_size
fsize = info.file_size
print "someimage compressed size is : %s" % csize
print "someimage real file size is : %s" % fsize
print """
So, knowing how zipfile.read() is actually implemented,
total needed ram is : %s
""" % (csize + fsize)

print "Well... Do I have that much memory available ???"

The python zipfile module is obviously broken...

s/is obviously broken/could be improved to handle huge files/

Making such statements may not be the best way to make friends...

Any advise?

Yes : Python is free software ('free' as in 'free speach' *and* as in
'free beer'), mostly written by benevolent contributors. So try and
improve the zipfile module by yourself, and submit your enhancements.
Then we all will be very grateful, and your name will be forever in the
Python Hall of Fame.

Or choose to behave as a whiny-whiny clueless luser making dumb
statements, and your name will be buried for a long long time in a lot
of killfiles.

It's up to you !-)

NB : If you go the first route, this may help:
http://www.python.org/doc/2.4.2/lib/module-zlib.html
with particular attention to the decompressobj.

HTH

 

[jordan.taylor2]

Take a look at the pywin32 extension, which I believe has some lower
level memory allocation and file capabilities that might help you in
this situation. If I'm completely wrong, someone please tell me XD.
Of course, you could just make the read() a step process, reading, O
lets say 8192 bytes at a time (could be bigger if u want), writes them
to the new file, and then reads the next portion. This will be slower
(not sure how much) than if you had some AMD X2 64 with 3 gigs of ram
and could just read the file all at once, but it should work.

 

[bruno at modulix]

Take a look at the pywin32 extension, which I believe has some lower
level memory allocation and file capabilities that might help you in
this situation.

But then the solution would not be portable, which would be a shame
since the zlib module (on which ZipFile relies for compression /
decompression) already has everything needed to handle streams.

 

[Sion Arrowsmith]

import zipfile
zip=zipfile.ZipFile('d:\somepath\cdimage.zip')
zip.namelist()
['someimage.iso'] [ ... ]
B) content=zip.read('someimage.iso')

Traceback (most recent call last):
File "<stdin>", line 1, in ?
File "D:\u\Python24\lib\zipfile.py", line 357, in read
bytes = dc.decompress(bytes)
MemoryError

I thought python was supposed to handle memory for you?

Yes, but it can't handle more memory than the operating system
is prepared to give it. How big is cdimage.zip? How big is the
uncompressed someimage.iso? How much memory do you have?

The python zipfile module is obviously broken...

This isn't at all obvious to me.

 

[Sion Arrowsmith]

http://mail.zope.org/pipermail/zope/2004-October/153882.html
"""
MemoryError is raised by Python when an underlying (OS-level) allocation
fails.
(...)
Normally this would mean that you were out of even virtual memory
(swap), but it could also be a symptom of a libc bug, a bad RAM chip, etc.
"""

There's another possibility, which I ran into recently. Which is a
problem with physical+virtual memory exceding the space addressable by
a process. So I've got 2G physical + 4G swap and I'm getting just such
a memory error -- I'm sure the compressed + uncompressed data isn't
going to eat all of that. But on a 32bit OS, it doesn't need to of
course. 2G is quite enough to cause problems....

 

[bruno at modulix]

This isn't at all obvious to me.

zipfile.read() does not seem to take full advantage of zlib's
decompressobj's features. This could perhaps be improved (left as an
exercice to the OP, who is obviously very good at detecting broken
memory management <g>).

Also, there's a known bug with file headers beginning past 2+GB - which
is not a very common case...
http://sourceforge.net/tracker/index.php?func=detail&aid=1189216&group_id=5470&atid=105470

So yes, there is actually something broken - but this has nothing to do
with the OP problem - *and* there are actually some limitions (FWIW, the
main goal of zipfile was mostly to implement support for zipped python
packages, not to replace Winzip). But what, the OP is going to fix this,
isn't he ?-)

 

[Roger Miller]

The basic problem is that the zipfile interface only reads and writes
whole files, so it may perform poorly or fail on huge files. At one
time I implemented a patch to allow reading files in chunks. However I
believe that the current interface has too many problems to solve by
incremental patching, and that a zipfile2 module is probably warranted.
(Is anyone working on this?)

In the meantime I think the best solution is often to just run an
external zip/unzip utility to do the heavy lifting.

 

[Bruno Desthuilliers]

Roger Miller a écrit :

The basic problem is that the zipfile interface only reads and writes
whole files, so it may perform poorly or fail on huge files. At one
time I implemented a patch to allow reading files in chunks. However I
believe that the current interface has too many problems to solve by
incremental patching,

Yeps, that was the general tone of some thread on python-dev. And from
what I saw of the source code, it may effectively not be the cleanest
part of the stdlib. But what, it does what it was written for at first :
provide working support for zipped packages.

and that a zipfile2 module is probably warranted.
(Is anyone working on this?)

Seems like Bob Ippolito was on the rank, but I guess you'll get better
answers on python-dev.

In the meantime I think the best solution is often to just run an
external zip/unzip utility to do the heavy lifting.

Indeed !-)

But while having zip/unzip installed OOTB on a unix-like system is close
to warrented, it may not be the case on Windows.

 

[Fredrik Lundh]

Is it me or is having to use os.system() all the time symtomatic of a
deficiency/things which are missing from python as a language?

it's you.

</F>

 

[Fredrik Lundh]

I take it that it's still a work in progress to be able to pythonify
everything, and until then we're just gonna have to rely on shell and
those great C coded coreutils and stuff like that. Ok, I'm rather fond
of Bash+coreutils, highest ratio of code lines to work I've ever
seen.... it's the real strength of Linux. Shame about Windows...

you make very little sense.

</F>

 

[Fredrik Lundh]

I've seen people using everything from zip to touch, either out of
laziness or out of the fact it wouldn't work very well in python, this
zip case is a good example.

so based on a limitation in one library, and some random code you've
seen on the internet, you're making generalizations about the language ?

the zip case is a pretty lousy example, btw; after all, using the
existing API, it's not that hard to implement an *incremental* read
function if the provided read-into-string version isn't sufficient:

import zipfile, zlib

##
# Given a 'zip' instance, copy data from the 'name' to the
# 'out' stream.

def explode(out, zip, name):

zinfo = zip.getinfo(name)

if zinfo.compress_type == zipfile.ZIP_STORED:
decoder = None
elif zinfo.compress_type == zipfile.ZIP_DEFLATED:
decoder = zlib.decompressobj(-zlib.MAX_WBITS)
else:
raise zipfile.BadZipFile("unsupported compression method")

zip.fp.seek(zinfo.file_offset)

size = zinfo.compress_size

while 1:
data = zip.fp.read(min(size, 8192))
if not data:
break
size -= len(data)
if decoder:
data = decoder.decompress(data)
out.write(data)

if decoder:
out.write(decoder.decompress('Z'))
out.write(decoder.flush())

</F>