print or die

[it_says_BALLS_on_your forehead]

I've recently come across a problem where I am parsing some large logs,
and end up filling up a file system. The process proceeds however,
since I don't check if the print succeeded or not. I created a simple
script to check print. The last two lines were executed if I didn't
have the 'or die...' clause in the for loop, and they were not if i
did:

use strict; use warnings;

my $time_start = localtime();

my $file = "big_file.txt";
open my $fh_out, '>', $file or die "can't open $file: $!\n";
for my $i ( 0 .. 1_000_000 ) {
print $fh_out 'A' x 100, "\n" or die "couldn't print: $!\n";
}
close $fh_out;

my $time_end = localtime();

print "$0: DONE.\n";
print "START: $time_start\n END: $time_end\n";

 

[it_says_BALLS_on_your forehead]

I've recently come across a problem where I am parsing some large logs,
and end up filling up a file system. The process proceeds however,
since I don't check if the print succeeded or not. I created a simple

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I meant to say 'die if the print failed'.

script to check print. The last two lines were executed if I didn't
have the 'or die...' clause in the for loop, and they were not if i
did:

i should also add that i tested this on a small filesystem where the
code below did indeed end up creating a file that maxed out the
filesystem.

 

[Uri Guttman]

isBoyf> I've recently come across a problem where I am parsing some
isBoyf> large logs, and end up filling up a file system. The process
isBoyf> proceeds however, since I don't check if the print succeeded
isBoyf> or not. I created a simple script to check print. The last two
isBoyf> lines were executed if I didn't have the 'or die...' clause in
isBoyf> the for loop, and they were not if i did:

isBoyf> my $file = "big_file.txt";
isBoyf> open my $fh_out, '>', $file or die "can't open $file: $!\n";
isBoyf> for my $i ( 0 .. 1_000_000 ) {
isBoyf> print $fh_out 'A' x 100, "\n" or die "couldn't print: $!\n";
isBoyf> }

isBoyf> However, I never see anyone check print. Is there a reason
isBoyf> this is not a good practice?

because about the only error print can generate is when the file system
is full and that isn't common enough to warrant checking all print
calls. if you are concerned enough about printing working then you
should check it.

uri

 

[it_says_BALLS_on_your forehead]

because about the only error print can generate is when the file system
is full and that isn't common enough to warrant checking all print
calls. if you are concerned enough about printing working then you
should check it.

great, thanks uri.

 

[xhoster]

I've recently come across a problem where I am parsing some large logs,
and end up filling up a file system. The process proceeds however,
since I don't check if the print succeeded or not. I created a simple
script to check print. The last two lines were executed if I didn't
have the 'or die...' clause in the for loop, and they were not if i
did:

In my hands, some fair number of "print"s report themselves as having
succeeded, even when they in fact did not, before they start returning
failure.

On the other hand, close always reported failure if a failure actually
did occur--even if every print (falsely) reported success.

....

for my $i ( 0 .. 1_000_000 ) {
print $fh_out 'A' x 100, "\n" or die "couldn't print: $!\n";
}
close $fh_out;

You must check the success of the close. Otherwise, you very well
may miss errors....

However, I never see anyone check print. Is there a reason this is not
a good practice?

....and once you do check the success of close, it is rarely important to
check the success of every print. If you don't want to wait for the close
to get the error, you can check print; But since you must check the close
anyway, checking the print is important only for perfomance, not
correctness.


$ perl -le '$|=1; open my $fh, ">/var/tmp/foo" or die $!;print $fh "foo" \
or die "$_ $!" foreach 1..10000;close $fh or die "close $!"'
1025 No space left on device at -e line 1.

So the first 1024 print attempts "succeeded"

Yet:

$ ls -lrt /var/tmp/foo
-rw-rw-r-- 1 xho Users 0 May 24 11:41 /var/tmp/foo

Whoops they didn't truly succeed.



Xho

 

[it_says_BALLS_on_your forehead]

In my hands, some fair number of "print"s report themselves as having
succeeded, even when they in fact did not, before they start returning
failure.

On the other hand, close always reported failure if a failure actually
did occur--even if every print (falsely) reported success.

that is interesting. one issue i've had with checking close was if i
opened a pipe to a large gzipped file, and last'd out of it
prematurely. the buffer was still full (i think this was the problem),
and i ended up getting an error on the close. i have a meeting now, but
will try to post a short script with the output here later.

...

You must check the success of the close. Otherwise, you very well
may miss errors....


...and once you do check the success of close, it is rarely important to
check the success of every print. If you don't want to wait for the close
to get the error, you can check print; But since you must check the close
anyway, checking the print is important only for perfomance, not
correctness.


$ perl -le '$|=1; open my $fh, ">/var/tmp/foo" or die $!;print $fh "foo" \
or die "$_ $!" foreach 1..10000;close $fh or die "close $!"'
1025 No space left on device at -e line 1.

So the first 1024 print attempts "succeeded"

Yet:

$ ls -lrt /var/tmp/foo
-rw-rw-r-- 1 xho Users 0 May 24 11:41 /var/tmp/foo

Whoops they didn't truly succeed.

that is very bizarre.

 

[it_says_BALLS_on_your forehead]

You must check the success of the close. Otherwise, you very well
may miss errors....

earlier i mentioned that checking close gave me problems when applied
to filehandles on pipes to large gzipped files. here is a sample:

my $file =
'/webstatmmk1/pre/febseclogs/rawLogs/quote.relay888.060321.rr111f.gz';

open my $fh_in, "gzip -dc $file | head -1000 |" or die "can't open
'$file': >>$!<<\n";
my $itr = 0;
while ( <$fh_in> ) {
last if ++$itr > 999;
print $itr, "\n";
}
close $fh_in or warn "can't close '$file': >>$!<<\n";

print "$0 done\n";


__RESULTS__
....
998
999
../check_close.pl done

now if i change the 999 to 10, i get:

__RESULTS__
1
2
3
4
5
6
7
8
9
10
Broken Pipe
can't close
'/webstatmmk1/pre/febseclogs/rawLogs/quote.relay888.060321.rr111f.gz':../check_close.pl done


if i had died instead of warned, then obviously all the code after the
close would not have been executed. i think it's reasonable to expect
to be able to last out of a loop such as this, and have that be valid.
the problem now is that i don't know if the close error is valid or
invalid. if it's a valid error, i would want to die. but if it's a
spurious error, i would not want to die.

 

[Charles DeRykus]

I've recently come across a problem where I am parsing some large logs,
and end up filling up a file system. The process proceeds however,
since I don't check if the print succeeded or not. I created a simple

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
I meant to say 'die if the print failed'.

script to check print. The last two lines were executed if I didn't
have the 'or die...' clause in the for loop, and they were not if i
did:

i should also add that i tested this on a small filesystem where the
code below did indeed end up creating a file that maxed out the
filesystem.

use strict; use warnings;

my $time_start = localtime();

my $file = "big_file.txt";
open my $fh_out, '>', $file or die "can't open $file: $!\n";
for my $i ( 0 .. 1_000_000 ) {
print $fh_out 'A' x 100, "\n" or die "couldn't print: $!\n";
}
close $fh_out;

my $time_end = localtime();

print "$0: DONE.\n";
print "START: $time_start\n END: $time_end\n";

Either that or you could just check the `close` which almost everyone
forgets too. The `Perl Cookbook` has a very good cautionary note:


These implicit closes are for convenience, not stability, because
they don't tell you whether the system call succeeded or failed.
Not all closes succeed. Even a close on a read-only file can fail.
For instance, you could lose access access to the device because
of a network outage. It's even more important to check the close
if the file was opened for writing. Otherwise you wouldn't notice
if the disk filled up.

I stumbled on this once too -- a critical function broke because of a
full partition. However, checking 'close' (which I didn't do) would
have generated:, ie., ENOSPC "No space left on device".

 

[xhoster]

that is interesting. one issue i've had with checking close was if i
opened a pipe to a large gzipped file, and last'd out of it
prematurely. the buffer was still full (i think this was the problem),
and i ended up getting an error on the close. i have a meeting now, but
will try to post a short script with the output here later.

If you close the reading end of a pipe early, the writing end (zcat, here)
will recieve a SIGPIPE and (depending on your system and zcat program) will
catch that signal or not, and if it catches it it will then print an STDERR
message about broken pipe or not, and exit with a non-zero value or not.
So if this is important, you need to check $! and possibly $? after the
close fails and choose what to do depending on what you see there.

(on my favorite linux system, closing a read pipe from zcat early gives an
empty $! and 13 (e.g. SIGPIPE) in $?), so I could do:

close $fh or $! and die "$! $?";

Not this will likely be a problem in your case. If you don't know whether
any given close statement is going to be closing a reading pipe vs a
writing file, you probably have bigger problems than checking the return
value of close!

DWIM is nice, but it can't do everything. Sometimes I don't even know what
I mean, much less expecting Perl to know.

that is very bizarre.

I don't think it is bizarre. Modern OS have umpteen layers of buffering.
Perl's buffering was turned off, and it printed into the kernel/fs
in-memory buffer fine, so as far as "print" is concerned the print
succeeded. It is only when the file is closed at the fs level that the
kernel/fs realizes it cannot be flushed to disk and reports its error back
to Perl.

Xho

 

[xhoster]

earlier i mentioned that checking close gave me problems when applied
to filehandles on pipes to large gzipped files. here is a sample:

my $file =
'/webstatmmk1/pre/febseclogs/rawLogs/quote.relay888.060321.rr111f.gz';

open my $fh_in, "gzip -dc $file | head -1000 |" or die "can't open
'$file': >>$!<<\n";
my $itr = 0;
while ( <$fh_in> ) {
last if ++$itr > 999;
print $itr, "\n";
}
close $fh_in or warn "can't close '$file': >>$!<<\n";

print "$0 done\n";
....

now if i change the 999 to 10, i get: ....
9
10
Broken Pipe
can't close
'/webstatmmk1/pre/febseclogs/rawLogs/quote.relay888.060321.rr111f.gz':
./check_close.pl done

if i had died instead of warned, then obviously all the code after the
close would not have been executed. i think it's reasonable to expect
to be able to last out of a loop such as this,

So do I. I also think it is reasonable to expect there to be consequences
from doing so

and have that be valid.
the problem now is that i don't know if the close error is valid or
invalid.

Neither does Perl! If you want to make highly robust programs, you have to
make the (sometimes substantial) effort to make them highly robust.

if it's a valid error, i would want to die. but if it's a
spurious error, i would not want to die.

Yep, and only you can know for sure which ones are spurious and which ones
are not.

Before I posted this:
close $fh or $! and die "$! $?";

But this will succeed if your piping programs gets blown out of the water
by some signal other than SIGPIPE. So if you are really paranoid (and
again, only you can know how paranoid you want to be, Perl can't know
that):

close $fh or ($?==13 and not $!) or die "$! $?"

I usually take the pragmatically paranoid approach--I generally start out
with some variant on close or die "$! $? $@" (unless I know from the
beginning that I want something more specific) and then evaluate and turn
off spurious dies as they arise.

Actually, I generally start it with 'or die', then kick myself for not
doing the 'or die "$! $? $@"' from the start.

Xho

 

[it_says_BALLS_on_your forehead]

(e-mail address removed) wrote:

I usually take the pragmatically paranoid approach--I generally start out
with some variant on close or die "$! $? $@" (unless I know from the
beginning that I want something more specific) and then evaluate and turn
off spurious dies as they arise.

this sounds like a great approach. i think i'll steal it.

 

[Charles DeRykus]

earlier i mentioned that checking close gave me problems when applied
to filehandles on pipes to large gzipped files. here is a sample:

my $file =
'/webstatmmk1/pre/febseclogs/rawLogs/quote.relay888.060321.rr111f.gz';

open my $fh_in, "gzip -dc $file | head -1000 |" or die "can't open
'$file': >>$!<<\n";
my $itr = 0;
while ( <$fh_in> ) {
last if ++$itr > 999;
print $itr, "\n";
}
close $fh_in or warn "can't close '$file': >>$!<<\n";

print "$0 done\n";


__RESULTS__
...
998
999
./check_close.pl done

now if i change the 999 to 10, i get:

__RESULTS__
1
2
3
4
5
6
7
8
9
10
Broken Pipe
can't close
'/webstatmmk1/pre/febseclogs/rawLogs/quote.relay888.060321.rr111f.gz':
./check_close.pl done


if i had died instead of warned, then obviously all the code after the
close would not have been executed. i think it's reasonable to expect
to be able to last out of a loop such as this, and have that be valid.
the problem now is that i don't know if the close error is valid or
invalid. if it's a valid error, i would want to die. but if it's a
spurious error, i would not want to die.

I played with this briefly and saw the "broken pipe" error as well.
However, there were no errors without `head -10` in the pipeline.
I kinda wonder if there's a timing problem which occurs because
`head -10` finishes quickly and closes its read buffer too fast
while gzip is still sending. Just a wild, totally unsubstantiated guess...

At any rate, couldn't you eliminate the `head` and roll your own
filtering inside the loop for the needed line count...

hth,

 

[xhoster]

....

I played with this briefly and saw the "broken pipe" error as well.

I saw it on some systems (SunOS) but not on others (Linux).

However, there were no errors without `head -10` in the pipeline.
I kinda wonder if there's a timing problem which occurs because
`head -10` finishes quickly and closes its read buffer too fast
while gzip is still sending. Just a wild, totally unsubstantiated
guess...

I think it is that when two pipes are involved, like with the head, perl
starts a shell and the shell starts the pipeline. Without the second pipe,
perl starts gzip directly. The shell (on some systems) complains to STDERR
when it detects a broken pipe, while gzip silently exits.

So with gzip, I get a $? of 13, while with head (or anything else causing
a shell to be fired up) I get a 36096, which I am guessing mean the shell
captures the SIGPIPE (36096 & 127 =0, meaning it didn't die of a signal)
and then turns around and exits with 36096 >> 8 = 141. Curiously,
141 & 127 = 13, so I think that by exiting with 141 the shell it trying to
tell you that the thing it called exited on signal 13.

At any rate, couldn't you eliminate the `head` and roll your own
filtering inside the loop for the needed line count...

I think his code was just illustrative, not the real case.

Xho

 

[Charles DeRykus]

I saw it on some systems (SunOS) but not on others (Linux).


I think it is that when two pipes are involved, like with the head, perl
starts a shell and the shell starts the pipeline. Without the second pipe,
perl starts gzip directly. The shell (on some systems) complains to STDERR
when it detects a broken pipe, while gzip silently exits.

So with gzip, I get a $? of 13, while with head (or anything else causing
a shell to be fired up) I get a 36096, which I am guessing mean the shell
captures the SIGPIPE (36096 & 127 =0, meaning it didn't die of a signal)
and then turns around and exits with 36096 >> 8 = 141. Curiously,
141 & 127 = 13, so I think that by exiting with 141 the shell it trying to
tell you that the thing it called exited on signal 13.

Thanks, the intervention of the shell gumming up the works certainly
sounds plausible.

 

[Charles DeRykus]

Thanks, the intervention of the shell gumming up the works certainly
sounds plausible.

Ah, a "talk-to-yourself" fork eliminates the additional shell as well
as the SIGPIPE that afflicts Solaris:

my $pid = open( my $fh_in, "-|" );
die "can't fork: $!" unless defined $pid;
unless ($pid) {
exec "gzip -dc $file | head -10" or die "exec failed: $!";
} else {
...
waitpid( $pid, 0 );
warn "reap error: $?" if $? and $? != -1;
}

--
Charles DeRykus






and the SIGPIPE

 

[Peter J. Holzer]

In my hands, some fair number of "print"s report themselves as having
succeeded, even when they in fact did not, before they start returning
failure. [...]
$ perl -le '$|=1; open my $fh, ">/var/tmp/foo" or die $!;print $fh
"foo" \
or die "$_ $!" foreach 1..10000;close $fh or die "close $!"'
1025 No space left on device at -e line 1.

So the first 1024 print attempts "succeeded"

[but file has length 0]

I don't think it is bizarre.

I did, until I spotted the error in your script.

Modern OS have umpteen layers of buffering. Perl's buffering was
turned off,

No, it wasn't. The $|=1 is in the wrong place. You need to insert

select $fh; $| = 1;

after the open.

and it printed into the kernel/fs in-memory buffer fine, so as far as
"print" is concerned the print succeeded.

But the OS should know that the disk is full and reject any attempts to
allocate new blocks.

It is only when the file is closed at the fs level that the kernel/fs
realizes it cannot be flushed to disk and reports its error back to
Perl.

This is not what you observed: You got an error after 1024 successful
writes, not after 10000.

So, why did your script "succeed" in writing 4096 bytes? Because that is
perls (or the stdio's) buffer size. When that buffer was full, perl
tried to flush it to disk and got an ENOSPC error. If you set $fh to
unbuffered, perl will get an error at the first print.

hp

 

[xhoster]

(e-mail address removed) wrote:
In my hands, some fair number of "print"s report themselves as
having succeeded, even when they in fact did not, before they start
returning failure. [...]
$ perl -le '$|=1; open my $fh, ">/var/tmp/foo" or die $!;print $fh
"foo" \
or die "$_ $!" foreach 1..10000;close $fh or die "close $!"'
1025 No space left on device at -e line 1.

So the first 1024 print attempts "succeeded"

[but file has length 0]

I don't think it is bizarre.

I did, until I spotted the error in your script.

Modern OS have umpteen layers of buffering. Perl's buffering was
turned off,

No, it wasn't. The $|=1 is in the wrong place. You need to insert

select $fh; $| = 1;

after the open.

Ah, good catch. Thanks.

But the OS should know that the disk is full and reject any attempts to
allocate new blocks.

And apparently in this case it does. But I still wouldn't depend on it
under all conditions! (Un)fortunately I only have one filesystem that is
completely full, so I can't test it on other types (like NFS-mounted ones.)

Xho

 

[Peter J. Holzer]

I played with this briefly and saw the "broken pipe" error as well.
However, there were no errors without `head -10` in the pipeline.
I kinda wonder if there's a timing problem which occurs because
`head -10` finishes quickly and closes its read buffer too fast
while gzip is still sending.

Almost right. It is not a timing problem: It doesn't matter whether head
finishes "quickly" or slowly. It is a matter of the *amount* of data:
head reads 10 lines and then exits. If gzip tries to write more than 10
lines, it will be terminated with a "broken pipe" signal.

(Actually the situation is a little more complicated, as gzip, head, and
the OS do some buffering, but basically, that's it)

hp

 

[Charles DeRykus]

Almost right. It is not a timing problem: It doesn't matter whether head
finishes "quickly" or slowly. It is a matter of the *amount* of data:
head reads 10 lines and then exits. If gzip tries to write more than 10
lines, it will be terminated with a "broken pipe" signal.

(Actually the situation is a little more complicated, as gzip, head, and
the OS do some buffering, but basically, that's it)

Yes, thanks, you're right. I meant to imply that the reader was closed
when there was still data coming from the writer which'll generate a
SIGPIPE. I was simply guessing that there might be a timing problem...
`head` with more lines appeared to work -- only with `head` grabbing a
few lines did the problem occur. (Of course, when I retried today on
Solaris, I saw saw no failures...even with a few lines. Sigh)

Anyway, Xho's analysis seems to indicate that the intervention of the
shell is somehow involved. If you fork and eliminate one of the shell
layers, the problem didn't occur so there's circumstantial evidence too
that might be the culprit.

 

[Peter J. Holzer]

[...]
Anyway, Xho's analysis seems to indicate that the intervention of the
shell is somehow involved. If you fork and eliminate one of the shell
layers, the problem didn't occur so there's circumstantial evidence too
that might be the culprit.

The problem (if you want to call killing a process after it has served
its purpose a problem) probably did occur just the same. But the error
message is printed by the shell. If you eliminate the shell, you would
have to check for the exit values of the child processes in the perl
script to detect that a child process was killed with a SIGPIPE.

hp