python crash problem

[Yelena]

I've been using python to run some large database recently.
I used dbfpy module to access some information stored in dbf format.
However, after 20hrs of running it gave me an error of :
Runtime Error!
This application has requested the Runtime to terminate it in an
unusual way.Please contact the application's support team for more
information.

The error popped out every time I ran them after 20 hrs. My prediction
was that
it ran over its memory usage, since it has been accumulating the
memory usage
when I was running the program. So I tried deleting/closing I/O
immediately
after they were processed. However, such changes did not make any
difference
in memory usage accumulation.

Could you please give me some advice in fixing problems like this?

Thank you

 

[Yelena]

Here is the code, I edited out some private info:

import sys
import urllib2
import string
import re
import datetime
from mx import DateTime
from dbfpy import dbf

addylist = ['add.dbf']
okset = ['P','L','I','B']
for addy in addylist:
db = dbf.Dbf(addy)
print (addy)
count = len(db)
db.close()
del db
foo = ((0, 1000),(1000,count/3), (count/3, count/3*2), (count/3*2,
count))
for (start,finish) in foo:
ab = dbf.Dbf(addy)
for x in range(start, finish):
print x
rec = ab[x]
if rec["flag"] in [1,3]:
INP_info = rec["address"] + " " + rec["city"] + " "
+rec["state"] + " " + rec["zip"]
INP_info = INP_info.replace(' ',',')
HOST_URL = '***'
KEY = '***'
STD_URL = '***'
STD_URL = STD_URL.replace('location=','location=' +
INP_info)
STD_URL = STD_URL.replace('key=','key=' + KEY)
Addy_INPUT = HOST_URL + STD_URL
try:
for line in urllib2.urlopen(Addy_INPUT):
safe = line
geoQ = safe.find('\"geocodeQualityCode\"',1)
geoCode = safe[(geoQ+22)geoQ+26)]
if geoCode[0] in okset:
lat = safe.find('\"lat\":', 1)
lng = safe.find('\"lng\":', 1)
lngbound = safe.find('\"mapUrl":', 1)
zipCode = safe.find('\"postalCode\"', 1)
zipbound = safe.find('\"adminArea1\"', 1)
rec["lat"] = float(safe[(lat + 6):
(lng-1)])
rec["long"] = float(safe[(lng+6):
(lngbound-2)])
rec["zip_final"] = safe[(zipCode+14):
(zipbound-2)]
rec["flag"] = 2
print ('ok ')
else:
rec["flag"] = 5
except:
rec["flag"] = 5
rec.store()
del rec
ab.close()
del ab

 

[Terry Reedy]

I've been using python

Which version?

> to run some large database recently.
I used dbfpy module to access some information stored in dbf format.

When having a problem with a 3rd party module, not part of the stdlib,
you should give a source.
http://sourceforge.net/projects/dbfpy/
This appears to be a compiled extension. Nearly always, when Python
crashes running such, it is a problem with the extension. So you
probably need to direct your question to the author or a project mailing
list if there is one.

 

[Emile van Sebille]

On 2/2/2011 11:19 AM Yelena said...

This application has requested the Runtime to terminate it in an
unusual way

This is a MS message -- did you look in the application event viewer?

Otherwise, you could try upping/reducing memory and confirm it dies
later/earlier. Perhaps there's an at or scheduled job killing off
python processes? How big is count? Does this job run for 20 hours
straight? Or is it launched periodically? Why the foo fudging? Just to
break the process into pieces? You could have the job log it's record
number after N hours (where N<20) and simply restart it...

Emile

 

[John Machin]

On 2/2/2011 2:19 PM, Yelena wrote:
..

When having a problem with a 3rd party module, not part of the stdlib,
you should give a source.
   http://sourceforge.net/projects/dbfpy/
This appears to be a compiled extension. Nearly always, when Python
crashes running such, it is a problem with the extension. So you
probably need to direct your question to the author or a project mailing
list if there is one.

It has always appeared to me to be a pure-Python package. There are
no .c or .pyx files in the latest source (.tgz) distribution. The
Windows installer installs only files whose extensions match
"py[co]?".