PyExcelerator

[tkpmep]

I write data to Excel files using PyExcelerator 0.6.3.a and have done
so successfully for small files (10-15 cells). I'm experiencing an
error when writing a big chunk of data (10,000 cells) to Excel. By way
of comparison, the same data writes perfectly well to a csv file using
Python's built in csv module. I run the program in PyScripter, and the
traceback shows the following sequence of calls:

main (my routine)
writeData (my routine)
save Line 563
get_biff_data Line 548
get_biff_data Line 1357
__row_blocks_rec Line 1276
get_cells_biff_data Line 200
get_biff_data Line 106

SystemError: frexp() result out of range

The line it stops at in get_biff_data is the line that starts with
packed =

def get_biff_data(self):
rk_encoded = 0

packed = struct.pack('<d', self.__number)

Any thoughts on what the problem could be?

Sincerely


Thomas Philips

 

[John Machin]

I write data to Excel files using PyExcelerator 0.6.3.a and have done
so successfully for small files (10-15 cells). I'm experiencing an
error when writing a big chunk of data (10,000 cells) to Excel. By way
of comparison, the same data writes perfectly well to a csv file using
Python's built in csv module. I run the program in PyScripter, and the
traceback shows the following sequence of calls:

main (my routine)
writeData (my routine)
save Line 563
get_biff_data Line 548
get_biff_data Line 1357
__row_blocks_rec Line 1276
get_cells_biff_data Line 200
get_biff_data Line 106

Grumble: When you need to post a traceback, please supply one that shows
on each line which *file* the source line is in. In this case there are
multiple files, and 9 classes spread over 4 files have a method called
get_biff_data.

SystemError: frexp() result out of range

The line it stops at in get_biff_data is the line that starts with
packed =

def get_biff_data(self):
rk_encoded = 0

packed = struct.pack('<d', self.__number)

Any thoughts on what the problem could be?

It is calling struct.pack (see structmodule.c in the Python source) to
encode one of your cell values as an 8-byte little-endian IEEE 754
floating-point number. This task is handed off to _PyFloat_Pack8() (see
floatobject.c). Here's the relevant bit:

"""
if (x < 0) {
sign = 1;
x = -x;
}
else
sign = 0;

f = frexp(x, &e);

/* Normalize f to be in the range [1.0, 2.0) */
if (0.5 <= f && f < 1.0) {
f *= 2.0;
e--;
}
else if (f == 0.0)
e = 0;
else {
PyErr_SetString(PyExc_SystemError,
"frexp() result out of range");
return -1;
}
"""
where x is your cell value as a C double.

frexp() is a standard C library routine. The above code is just checking
that it has done its job correctly. AFAICT from reading K&R v2 p251, the
code is correct.

Aside: isn't open source wonderful? Well it is for me. It could be
wonderful for you too

I'm not an IEEE 754 guru by any means, so I'm guessing your value could
be an Inf (infinity) or a NaN (not a number) -- there's no explicit test
for those in the code -- or you have stumbled on a value that's a corner
case on your platform. Maybe it's -0.0 (zero with the sign bit set).

Grumble: When you have a problem like this, you should state what
version of Python you are using, on what platform. The line you get when
you run Python at the command line is a start e.g.
"""Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)]
on win32"""

Please tell us this, plus explicitly what OS and what hardware you are
running on.

OK, now what's this dodgy number?? Unfortunately, the error is happening
right at the end of your process, in the workbook save method. The
only info about which row/column is buried deep in pyExcelerator. We
don't (yet) want to start poking about there ...

Try these steps:
1. Search your alternative CSV output file for the text "Inf" or "NaN"
(case insensitive).
2. Put some code in your cell write routine to filter the values you are
writing:

absval = abs(value)
SMALL = 0.000001 # depends on what is reasonable for your app
LARGE = 10.0 ** 10 # depends ...
if not (value == 0.0 or (SMALL <= absval <= LARGE)):
print "strange value %r in row=%d col=%d" % (value, rownum, colnum)

Then come back and tell us what if anything you've found.

HTH,
John