python and os.system() failure

[kennywiggin]

Hi,

I'm actually not sure where the error is actually occurring, but the
effects are easily reproduced, but not reproducable. By not
reproducable, I mean the error does not occur exactly in the same
"spot" but it always happens.

I have my CD collection ripped to flac, so I have >4000 flacs. I wrote
this script that will automatically encode the flacs into mp3 or ogg
while transferring all the metaflac info over to id3 tag of the mp3.

Generally, the script works like this:

Get list of flac
for each flac file:
Store metaflac_info in an information class
os.system(decode flac using /usr/bin/flac)
os.system(encode mp3 and embed info stored in class)

Now, this script will go on for some time but after 40-100 files the
decode will fail with /usr/bin/flac reporting an error in reading the
flac file.

..flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME

The script fails to decode any more flac files until the vm
crashes from having too many open files.

The flac file is perfectly fine, though. Restarting the script, the
flac file that "crashed" decodes/encodes just fine. Analysis by 'flac
-t' or 'flac -a' show a correct flac file. How ever, if I try to
decode a "bad" flac file in parallel with the script after the script
has "failed", I can't decode the flac and get the same error. Yet, I
can do 'flac -a' or 'flac -t' on the file and it reports 100% good in
parallel to the "dead" script.

It is not a disk space issue as I've checked. No stray tempfiles are
created since the command to create them fails when opening the flac
file.

I have tried invoking garbage collection after a set number of file
and sleeping the thread after running into the error, but neither
worked. The most likely culprit is my script, but it seems too simple
to "die" like that. I don't think it's a problem with the 'flac'
application. Another possible culprit is a python VM issue that I am
not aware of.

Any advice would be appreciated,

Kenny

Debian Linux, i686
Python 2.3 (2.3.5-2) and earlier versions of 2.3
flac 1.1.1 (1.1.1-5)

 

[Michael Hoffman]

.flac: ERROR while decoding data
state = FLAC__STREAM_DECODER_READ_FRAME

The script fails to decode any more flac files until the vm
crashes from having too many open files.

Whoa! Time to start using file.close() after you're done with each
file.

If that doesn't fix your problem (perhaps you are running out of file
handles?), I'd run strace on the flac invocation and see how a good one
differs from a bad one.

 

[Anthony]

Thanks for the tip on strace. I will look into it.

I only have a tempfile object:

tmp = tempfile.mkstemp(dir="/var/tmp/")

I will try to explicitly call a close() or whatever the syntax is to
finalize the object to deletion by garbage collection.

Thanks,

Kenny

 

[Michael Hoffman]

I only have a tempfile object:

tmp = tempfile.mkstemp(dir="/var/tmp/")

Why aren't you using NamedTemporaryFile instead? Using mkstemp adds a
lot of complications that are usually unnecessary.

I will try to explicitly call a close() or whatever the syntax is to
finalize the object to deletion by garbage collection.

Files created with mkstemp are neither closed nor deleted when they are
finalized. I don't see how GC comes into it either.

 

[Anthony]

Why aren't you using NamedTemporaryFile instead? Using mkstemp adds a

lot of complications that are usually unnecessary.

I believe I originally used mktemp(), but forgot how I learned that I
should use mkstemp() instead. I originally wrote the script over 1.5
years ago. =)
I may have been afraid of the "deleted when closed" at the time, but
now that really isn't an issue.

Files created with mkstemp are neither closed nor deleted when they are
finalized. I don't see how GC comes into it either.

Ah, that may explain it. Python.org did not say that mkstemp() is
outside of close or gc. I tried to both close and call gc at various
intervals, but, obviously, neither affected the running of the script.

Thanks alot, I think this will help.