Unix shell script with Perl one-liner causes error

[Simon O]

Hi there

I have this Unix Borne shell script that does something like this in a
loop :

for FILE in *
do
ROW_COUNT_HEADER=`...`
if [ ${ROW_COUNT_HEADER} -ne 0 ]
then
...
# 1) OLD Option with Sed :
# cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
# 2) NEW Option with Perl :
cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
file.dat
fi
bzip2 ${FILE}
mv ${FILE}.bz2 ${LATEST_ARCHIVE}/IGCPDATA/
done

I was having problems with using sed, so changed to option 2) using
perl which then gave a really weird error. It does what it's supposed
to but when it gets to the bzip / move section, it gives an error as
though it's still busy with ${FILE} ?? - being driving me MAD ! Almost
as though Perl is has said it's finished but is not ?

The error is this :

bzip2: I/O or other error, bailing out. Possible reason follows.
bzip2: No such file or directory
Input file = ABS_MTM_By_Trade, output file =
ABS_MTM_By_Trade.bz2
bzip2: Deleting output file ABS_MTM_By_Trade.bz2, if it exists.
bzip2: WARNING: deletion of output file (apparently) failed.

and of the 29 files in *, this error occurs randomly for only say five
of the files and the others are ok ??! The files are definitely there.

Any ideas ?

Thanks
Simon

 

[John W. Krahn]

I have this Unix Borne shell script that does something like this in a
loop :

for FILE in *
do
ROW_COUNT_HEADER=`...`
if [ ${ROW_COUNT_HEADER} -ne 0 ]
then
...
# 1) OLD Option with Sed :
# cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
# 2) NEW Option with Perl :
cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
file.dat

You don't need the /g option because /$/ only occurs once in every line.

Or you could do that like:

perl -F'\t' -lane'splice @F, 32; print join "\t", @F, $ARGV' ${FILE}

fi
bzip2 ${FILE}
mv ${FILE}.bz2 ${LATEST_ARCHIVE}/IGCPDATA/
done

I was having problems with using sed, so changed to option 2) using
perl which then gave a really weird error. It does what it's supposed
to but when it gets to the bzip / move section, it gives an error as
though it's still busy with ${FILE} ?? - being driving me MAD ! Almost
as though Perl is has said it's finished but is not ?

The error is this :

bzip2: I/O or other error, bailing out. Possible reason follows.
bzip2: No such file or directory
Input file = ABS_MTM_By_Trade, output file =
ABS_MTM_By_Trade.bz2
bzip2: Deleting output file ABS_MTM_By_Trade.bz2, if it exists.
bzip2: WARNING: deletion of output file (apparently) failed.

I don't know enough about bzip to be able to diagnose its errors.



John

 

[Simon O]

Thanks John for the alternatives.

Not sure that would solve the problem as the error seems to be in the
way the shell passes data to Perl and get's it back.

Bzip - could be any other command - gzip or mv, for that matter - seems
as though the file is locked by Perl ?

I thought that because it's unix | perl | unix I thought the file
handling would all be done by Unix ?

Simon

I have this Unix Borne shell script that does something like this in a
loop :

for FILE in *
do
ROW_COUNT_HEADER=`...`
if [ ${ROW_COUNT_HEADER} -ne 0 ]
then
...
# 1) OLD Option with Sed :
# cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
# 2) NEW Option with Perl :
cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
file.dat

You don't need the /g option because /$/ only occurs once in every line.

Or you could do that like:

perl -F'\t' -lane'splice @F, 32; print join "\t", @F, $ARGV' ${FILE}

fi
bzip2 ${FILE}
mv ${FILE}.bz2 ${LATEST_ARCHIVE}/IGCPDATA/
done

I was having problems with using sed, so changed to option 2) using
perl which then gave a really weird error. It does what it's supposed
to but when it gets to the bzip / move section, it gives an error as
though it's still busy with ${FILE} ?? - being driving me MAD ! Almost
as though Perl is has said it's finished but is not ?

The error is this :

bzip2: I/O or other error, bailing out. Possible reason follows.
bzip2: No such file or directory
Input file = ABS_MTM_By_Trade, output file =
ABS_MTM_By_Trade.bz2
bzip2: Deleting output file ABS_MTM_By_Trade.bz2, if it exists.
bzip2: WARNING: deletion of output file (apparently) failed.

I don't know enough about bzip to be able to diagnose its errors.



John

 

[Dr.Ruud]

Simon O schreef:

for FILE in *
do
cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g"
done

Probably unrelated to your problem, but the /g modifier is not right.
Same with the sed command. In both cases, the substitution is done per
line.

You don't need cut:

for FILE in *
do
perl -ple "s/^([^\t]*(?:\t[^\t]*){0,31}).*$/\$1\t$FILE/" \
"$FILE" >> file.dat
done

You can of course also do that shell-for-loop in Perl (the language).
You can even get rid of the shell-script and do it all with perl (the
implementation), which would give you better control over what goes
wrong.

 

[Mumia W.]

Hi there

I have this Unix Borne shell script that does something like this in a
loop :

for FILE in *
do
ROW_COUNT_HEADER=`...`
if [ ${ROW_COUNT_HEADER} -ne 0 ]
then
...
# 1) OLD Option with Sed :
# cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
# 2) NEW Option with Perl :
cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
file.dat

Perhaps you need to backslash escape that first '$':
s/\$/\t${FILE}/g

Or you could just print:
perl -nle "print; print q[ ${FILE}]"

[...]

 

[John W. Krahn]

Thanks John for the alternatives.

Not sure that would solve the problem as the error seems to be in the
way the shell passes data to Perl and get's it back.

Bzip - could be any other command - gzip or mv, for that matter - seems
as though the file is locked by Perl ?

No, Perl does not lock the file, and even if you had explicitly flock()ed the
file, locking files in Unix is only advisory. Besides which, in your example,
there is no physical file to lock because the data is being piped to perl from
another program.

I thought that because it's unix | perl | unix I thought the file
handling would all be done by Unix ?

Each process in the pipe, be it cut or sed or awk or tr or perl, etc., treats
the data the same way.


John

 

[Charles DeRykus]

Hi there

I have this Unix Borne shell script that does something like this in a
loop :

for FILE in *
do
ROW_COUNT_HEADER=`...`
if [ ${ROW_COUNT_HEADER} -ne 0 ]
then
...
# 1) OLD Option with Sed :
# cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
# 2) NEW Option with Perl :
cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
file.dat
fi
bzip2 ${FILE}
mv ${FILE}.bz2 ${LATEST_ARCHIVE}/IGCPDATA/
done

I was having problems with using sed, so changed to option 2) using
perl which then gave a really weird error. It does what it's supposed
to but when it gets to the bzip / move section, it gives an error as
though it's still busy with ${FILE} ?? - being driving me MAD ! Almost
as though Perl is has said it's finished but is not ?

The error is this :

bzip2: I/O or other error, bailing out. Possible reason follows.
bzip2: No such file or directory
Input file = ABS_MTM_By_Trade, output file =
ABS_MTM_By_Trade.bz2
bzip2: Deleting output file ABS_MTM_By_Trade.bz2, if it exists.
bzip2: WARNING: deletion of output file (apparently) failed.

and of the 29 files in *, this error occurs randomly for only say five
of the files and the others are ok ??! The files are definitely there.

Maybe you could wrap a sleep/retry loop around `bzip2 ${FILE}` in case
this is just a transient, recoverable error. This is unrelated to Perl
though so you might want to try one of the shell groups for followups.

hth,

 

[Simon O]

Gentlemen,

Thanks for all your helpful suggestions, I've justed joined this group
and am well impressed. I think I've now found a solution, though I
still don't understand why there was a problem.

To reduce rowcounts, for testing purposes, I had put a head in the
chain of piped commands as follows :

cat ${FILE} | cut -f1-32 | head | perl -ple "s/$/\t${FILE}/" >>
file.dat

When I took it out, it worked fine !?? Confused ..

Sure, my Perl is not great and as always with Perl there are loads of
ways of killing the cat, but it works !.

Thanks
Simon

 

[Mumia W.]

[...]
# 1) OLD Option with Sed :
# cat ${FILE} | cut -f1-32 | sed "s/$/ ${FILE}/g" >> file.dat
# 2) NEW Option with Perl :
cat ${FILE} | cut -f1-32 | perl -ple "s/$/\t${FILE}/g" >>
file.dat

Perhaps you need to backslash escape that first '$':
s/\$/\t${FILE}/g

And don't forget that your ${FILE} might contain slashes that conflict
with the s/// operator's slashes, so use another delimiter:

s,\$,\t${FILE},g

You can do something like this with sed also.