STDOUT pass-through with system()

[Justin]

Hi all,

I've inherited a rather scrappy setup whereby a perl script calls a 2nd
script with system() (which writes to STDOUT) which then exec's a command
(which also writes to STDOUT). Under v5.8.0 I lose the STDOUT (and
STDERR) from the second script onwards. Under our legacy v5.002 it works
as I would hope.

e.g. I have 3 scripts, stage1, stage2, stage3. I run stage1 from the
console, which calls stage2 with system(), which calls stage3 with exec.

-----------------stage1------------------
#!/usr/local/bin/perl
select STDOUT; $|=1; #autoflush STDOUT
print STDOUT "Stage 1 STDOUT\n";
print STDERR "Stage 1 STDERR\n";
#Now call Stage 2 with system() which forks, then retuns
print "Calling (system) Stage 2 from Stage 1...\n";
system("stage2");
print "Returned from Stage 2 to Stage 1\n";

------------------stage2-----------------
#!/usr/local/bin/perl
select STDOUT; $|=1; #autoflush STDOUT
print STDOUT "Stage 2 STDOUT\n";
print STDERR "Stage 2 STDERR\n";
#Now call Stage 3 with system() which forks, then retuns
print "Calling (via exec) Stage 3 from Stage 2...\n";
exec("stage3");
print "Returned from Stage 3 to Stage 2\n";

-------------------stage3----------------
#!/usr/local/bin/perl
select STDOUT; $|=1; #autoflush STDOUT
print STDOUT "Stage 3 STDOUT\n";
print STDERR "Stage 3 STDERR\n";



I run the commands from the console so I see STDOUT and STDERR. Under
Perl v5.002 I get the STDOUT (and STDERR) from all 3 stages displayed:

Stage 1 STDOUT
Stage 1 STDERR
Calling (system) Stage 2 from Stage 1...
Stage 2 STDOUT
Stage 2 STDERR
Calling (via exec) Stage 3 from Stage 2...
Stage 3 STDOUT
Stage 3 STDERR
Returned from Stage 2 to Stage 1


Under Perl v5.8.0 I get the following:

Stage 1 STDOUT
Stage 1 STDERR
Calling (system) Stage 2 from Stage 1...
Returned from Stage 2 to Stage 1


Now I want it to work like v5.002. You can see I've tried autoflush.
According to Perlfaq8, "With system(), both STDOUT and STDERR will go the
same place as the script's versions of these, unless the command
redirects them."

Any ideas?

Cheers,

Justin

 

[xhoster]

Hi all,

I've inherited a rather scrappy setup whereby a perl script calls a 2nd
script with system() (which writes to STDOUT) which then exec's a command
(which also writes to STDOUT). Under v5.8.0 I lose the STDOUT (and
STDERR) from the second script onwards. Under our legacy v5.002 it works
as I would hope.

e.g. I have 3 scripts, stage1, stage2, stage3. I run stage1 from the
console, which calls stage2 with system(), which calls stage3 with exec.

-----------------stage1------------------
#!/usr/local/bin/perl
select STDOUT; $|=1; #autoflush STDOUT
print STDOUT "Stage 1 STDOUT\n";
print STDERR "Stage 1 STDERR\n";
#Now call Stage 2 with system() which forks, then retuns
print "Calling (system) Stage 2 from Stage 1...\n";
system("stage2");
print "Returned from Stage 2 to Stage 1\n";

------------------stage2-----------------
#!/usr/local/bin/perl
select STDOUT; $|=1; #autoflush STDOUT
print STDOUT "Stage 2 STDOUT\n";
print STDERR "Stage 2 STDERR\n";
#Now call Stage 3 with system() which forks, then retuns
print "Calling (via exec) Stage 3 from Stage 2...\n";
exec("stage3");
print "Returned from Stage 3 to Stage 2\n";

-------------------stage3----------------
#!/usr/local/bin/perl
select STDOUT; $|=1; #autoflush STDOUT
print STDOUT "Stage 3 STDOUT\n";
print STDERR "Stage 3 STDERR\n";

I notice that you are not checking the success of your system/exec calls.
Shame on you. I bet you dollars to donuts that that is your problem.
Furthermore, I guess that your shebang line is wrong.


Because you start stage1 by doing:

perl stage1.pl

It will run, because it ignores the shebang line path. However, stage2
never starts in the first place, but because you don't check for success
of the system call, you don't know that.

Xho

 

[Justin]

I notice that you are not checking the success of your system/exec
calls. Shame on you. I bet you dollars to donuts that that is your
problem. Furthermore, I guess that your shebang line is wrong.

I am shamed. Indeed the system() was failing, however its not due to the
hashbang line. I wasn't specifying the full path to the scripts, and I was
assuming the pwd was in the path. Which it is on the old system but not the
new.

I should have pointed out that I start stage1 as "./stage1". Had I done
that I might have spotted my own mistake.

I take some solace in the fact that I wrote these test scripts and they
reproduced the error I was experiencing in my production system. Turns out
it must be two different causes producing similar errors since in my
production system I specify full paths to the executables.

Thanks for pointing out my error Xho. I hang my head. Sigh.

Justin

 

[Justin]

Turns out it must be two different causes producing similar errors
since in my production system I specify full paths to the executables.

A classy guy replying to my own post but just in case it ever turns out to
be useful to someone else...

It turns out my problem was a couple of debug characters weren't being
printed to STDERR but to the default STDOUT by mistake. In v5.002 these few
chars in the STDOUT buffer were disappearing during either the system() or
exec(), and since they'd never been seen in any output no-one spotted the
error (I don't know why this would happen mind you). Under v5.8.0 these
couple of chars were being flushed to STDOUT into the "data stream" which
was screwing up latter processing. So it looks like some strange behaviour
of v5.002 was covering up a bug that was there all the time...

 

[xhoster]

A classy guy replying to my own post but just in case it ever turns out
to be useful to someone else...

It turns out my problem was a couple of debug characters weren't being
printed to STDERR but to the default STDOUT by mistake. In v5.002 these
few chars in the STDOUT buffer were disappearing during either the
system() or exec(), and since they'd never been seen in any output no-one
spotted the error (I don't know why this would happen mind you). Under
v5.8.0 these couple of chars were being flushed to STDOUT into the "data
stream" which was screwing up latter processing. So it looks like some
strange behaviour of v5.002 was covering up a bug that was there all the
time...

I believe the behavior you observe is due to this:

(from perldoc -f system)
Beginning with v5.6.0, Perl will attempt to flush all files
opened for output before any operation that may do a fork,
but this may not be supported on some platforms (see
perlport). To be safe, you may need to set $| ($AUTOFLUSH
in English) or call the "autoflush()" method of "IO::Handle"
on any open handles.

Cheers,

Xho