Issue implementing Runtime.exec() with StreamGobbler

[Vic]

I am not very well conversant with java and have a very basic and
limited java experience being a QA engg. I am currently looking at a
code which isn't very consistent in our testing atuomation framework.
Here is the code ->
// executes system commands and captures STDOUT and STDERR
import java.io.*;
import java.util.*;

class StreamGobbler extends Thread {
InputStream is;
String type;
String result;

StreamGobbler(InputStream is, String type){
this.is = is;
this.type = type;
}

public void run(){
StringBuffer buffer = new StringBuffer();

try{
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while((line = br.readLine()) != null){
buffer.append(line);
buffer.append("\n");
}
}
catch(IOException ioe){
ioe.printStackTrace();
}
finally{
result = buffer.toString();
}
}

public String getResult(){
return result;
}
}

class NewCommandExecuter {
public String stdout = null;
public String stderr = null;
private String line;

public NewCommandExecuter (String cmd) {
try{
//VikP - This needed to be modified to incorporate the change in
syntax for executing getData.pl by Mark Silverman for PR#5026342
System.out.println("New Command Executer: EXECUTING COMMAND:
"+cmd);
String [] execmd = {"bash","-c",cmd};
//Process proc = Runtime.getRuntime().exec(new String[]{"bash","-
c",cmd});
Process proc = Runtime.getRuntime().exec(execmd);
//Process proc = Runtime.getRuntime().exec(cmd);
StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream
(),"ERROR");
StreamGobbler outputGobbler = new StreamGobbler
(proc.getInputStream(),"OUTPUT");

errorGobbler.start();
outputGobbler.start();

int val = proc.waitFor();
System.out.println("ExitValue for Command "+cmd+" is: "+val);

stdout = outputGobbler.getResult();
stderr = errorGobbler.getResult();

//VikP - Waits for these threads to die
errorGobbler.join();
outputGobbler.join();
//
}
catch (Throwable t){
t.printStackTrace();
}

if (stdout.equals(""))
stdout = null;

if (stderr.equals(""))
stderr = null;
}
}

I call the the above in a loop (of about 2000 cases) as ->
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);

// verity xml output
if (xmlOut.stdout == null)
throw new Exception ("Xml-out query '"+xmlOutCmd+"' didn't return
any results. STDERR returned:\n"+xmlOut.stderr);

// save output to a temp file
writeStringToFile(tempXmlFile, xmlOut.stdout); // save xml output
to a file

The problem that I am facing is it works fine but it doesn't go
through the entire loop (all the test cases we have) and almost always
gets terminated prematurely completing only a partial list of cases.
So that makes me wonder if there is anything wrong in the
StreamGobbler and/or NewCommandExecuter. I tried a bunch of things in
those classes but definitely I am missing something here. Please help.
Thanks

 

[Vic]

I am not very well conversant with java and have a very basic and
limited java experience being a QA engg. I am currently looking at a
code which isn't very consistent in our testing atuomation framework.
Here is the code ->
// executes system commands and captures STDOUT and STDERR
import java.io.*;
import java.util.*;

class StreamGobbler extends Thread {
    InputStream is;
    String type;
    String result;

    StreamGobbler(InputStream is, String type){
        this.is = is;
        this.type = type;
    }

    public void run(){
        StringBuffer buffer = new StringBuffer();

        try{
            InputStreamReader isr = new InputStreamReader(is);
            BufferedReader br = new BufferedReader(isr);
            String line = null;
            while((line = br.readLine()) != null){
                buffer.append(line);
                buffer.append("\n");
            }
        }
        catch(IOException ioe){
            ioe.printStackTrace();
        }
        finally{
            result = buffer.toString();
        }
    }

    public String getResult(){
        return result;
    }

}

class NewCommandExecuter {
    public String stdout = null;
    public String stderr = null;
    private String line;

    public NewCommandExecuter (String cmd) {
        try{
            //VikP - This needed to be modified to incorporate the change in
syntax for executing getData.pl by Mark Silverman for PR#5026342
            System.out.println("New Command Executer: EXECUTING COMMAND:
"+cmd);
            String [] execmd = {"bash","-c",cmd};
            //Process proc = Runtime.getRuntime().exec(new String[]{"bash","-
c",cmd});
            Process proc = Runtime.getRuntime().exec(execmd);
            //Process proc = Runtime.getRuntime().exec(cmd);
            StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream
(),"ERROR");
            StreamGobbler outputGobbler = new StreamGobbler
(proc.getInputStream(),"OUTPUT");

            errorGobbler.start();
            outputGobbler.start();

            int val = proc.waitFor();
            System.out.println("ExitValue for Command "+cmd+" is: "+val);

            stdout = outputGobbler.getResult();
            stderr = errorGobbler.getResult();

            //VikP - Waits for these threads to die
            errorGobbler.join();
            outputGobbler.join();
            //
        }
        catch (Throwable t){
                t.printStackTrace();
        }

        if (stdout.equals(""))
            stdout = null;

        if (stderr.equals(""))
            stderr = null;
    }

}

I call the the above in a loop (of about 2000 cases) as ->
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);

// verity xml output
            if (xmlOut.stdout == null)
                throw new Exception ("Xml-out query '"+xmlOutCmd+"' didn't return
any results. STDERR returned:\n"+xmlOut.stderr);

            // save output to a temp file
            writeStringToFile(tempXmlFile, xmlOut.stdout); // save xml output
to a file

The problem that I am facing is it works fine but  it doesn't go
through the entire loop (all the test cases we have) and almost always
gets terminated prematurely completing only a partial list of cases.
So that makes me wonder if there is anything wrong in the
StreamGobbler and/or NewCommandExecuter. I tried a bunch of things in
those classes but definitely I am missing something here. Please help.
Thanks

xmlOutCmd looks something like this -> /home/qauser/770currentLOADER/
bin/loader -c -u genevaman -w qa -f Transaction917.0.lrd -o
Transaction917.0.lrd.xml

Also the implementation of Runtime.exec() was developed with the help
from
http://www.javaworld.com/javaworld/jw-12-2000/jw-1229-traps.html?page=4

Thanks in advance

 

[Knute Johnson]

I am not very well conversant with java and have a very basic and
limited java experience being a QA engg. I am currently looking at a
code which isn't very consistent in our testing atuomation framework.
Here is the code ->
// executes system commands and captures STDOUT and STDERR
import java.io.*;
import java.util.*;

class StreamGobbler extends Thread {
InputStream is;
String type;
String result;

StreamGobbler(InputStream is, String type){
this.is = is;
this.type = type;
}

public void run(){
StringBuffer buffer = new StringBuffer();

try{
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while((line = br.readLine()) != null){
buffer.append(line);
buffer.append("\n");
}
}
catch(IOException ioe){
ioe.printStackTrace();
}
finally{
result = buffer.toString();
}
}

public String getResult(){
return result;
}
}

class NewCommandExecuter {
public String stdout = null;
public String stderr = null;
private String line;

public NewCommandExecuter (String cmd) {
try{
//VikP - This needed to be modified to incorporate the change in
syntax for executing getData.pl by Mark Silverman for PR#5026342
System.out.println("New Command Executer: EXECUTING COMMAND:
"+cmd);
String [] execmd = {"bash","-c",cmd};
//Process proc = Runtime.getRuntime().exec(new String[]{"bash","-
c",cmd});
Process proc = Runtime.getRuntime().exec(execmd);
//Process proc = Runtime.getRuntime().exec(cmd);
StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream
(),"ERROR");
StreamGobbler outputGobbler = new StreamGobbler
(proc.getInputStream(),"OUTPUT");

errorGobbler.start();
outputGobbler.start();

int val = proc.waitFor();
System.out.println("ExitValue for Command "+cmd+" is: "+val);

stdout = outputGobbler.getResult();
stderr = errorGobbler.getResult();

//VikP - Waits for these threads to die
errorGobbler.join();
outputGobbler.join();
//
}
catch (Throwable t){
t.printStackTrace();
}

if (stdout.equals(""))
stdout = null;

if (stderr.equals(""))
stderr = null;
}
}

I call the the above in a loop (of about 2000 cases) as ->
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);

// verity xml output
if (xmlOut.stdout == null)
throw new Exception ("Xml-out query '"+xmlOutCmd+"' didn't return
any results. STDERR returned:\n"+xmlOut.stderr);

// save output to a temp file
writeStringToFile(tempXmlFile, xmlOut.stdout); // save xml output
to a file

The problem that I am facing is it works fine but it doesn't go
through the entire loop (all the test cases we have) and almost always
gets terminated prematurely completing only a partial list of cases.
So that makes me wonder if there is anything wrong in the
StreamGobbler and/or NewCommandExecuter. I tried a bunch of things in
those classes but definitely I am missing something here. Please help.
Thanks

By 'terminated', do you mean it throws an exception or just ends without
running all the tests?

 

[Vic]

I am not very well conversant with java and have a very basic and
limited java experience being a QA engg. I am currently looking at a
code which isn't very consistent in our testing atuomation framework.
Here is the code ->
// executes system commands and captures STDOUT and STDERR
import java.io.*;
import java.util.*;

class StreamGobbler extends Thread {
    InputStream is;
    String type;
    String result;

    StreamGobbler(InputStream is, String type){
   this.is = is;
   this.type = type;
    }

    public void run(){
   StringBuffer buffer = new StringBuffer();

   try{
       InputStreamReader isr = new InputStreamReader(is);
       BufferedReader br = new BufferedReader(isr);
       String line = null;
       while((line = br.readLine()) != null){
           buffer.append(line);
           buffer.append("\n");
       }
   }
   catch(IOException ioe){
       ioe.printStackTrace();
   }
   finally{
       result = buffer.toString();
   }
    }

    public String getResult(){
   return result;
    }
}

class NewCommandExecuter {
    public String stdout = null;
    public String stderr = null;
    private String line;

    public NewCommandExecuter (String cmd) {
   try{
       //VikP - This needed to be modified to incorporate the change in
syntax for executing getData.pl by Mark Silverman for PR#5026342
       System.out.println("New Command Executer: EXECUTING COMMAND:
"+cmd);
       String [] execmd = {"bash","-c",cmd};
       //Process proc = Runtime.getRuntime().exec(new String[]{"bash","-
c",cmd});
       Process proc = Runtime.getRuntime().exec(execmd);
       //Process proc = Runtime.getRuntime().exec(cmd);
       StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream
(),"ERROR");
       StreamGobbler outputGobbler = new StreamGobbler
(proc.getInputStream(),"OUTPUT");

       errorGobbler.start();
       outputGobbler.start();

       int val = proc.waitFor();
       System.out.println("ExitValue for Command "+cmd+" is: "+val);

       stdout = outputGobbler.getResult();
       stderr = errorGobbler.getResult();

       //VikP - Waits for these threads to die
       errorGobbler.join();
       outputGobbler.join();
       //
   }
   catch (Throwable t){
           t.printStackTrace();
   }

   if (stdout.equals(""))
       stdout = null;

   if (stderr.equals(""))
       stderr = null;
    }
}

I call the the above in a loop (of about 2000 cases) as ->
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);

// verity xml output
       if (xmlOut.stdout == null)
           throw new Exception ("Xml-out query '"+xmlOutCmd+"' didn't return
any results. STDERR returned:\n"+xmlOut.stderr);

       // save output to a temp file
       writeStringToFile(tempXmlFile, xmlOut.stdout); // save xml output
to a file

The problem that I am facing is it works fine but  it doesn't go
through the entire loop (all the test cases we have) and almost always
gets terminated prematurely completing only a partial list of cases.
So that makes me wonder if there is anything wrong in the
StreamGobbler and/or NewCommandExecuter. I tried a bunch of things in
those classes but definitely I am missing something here. Please help.
Thanks

By 'terminated', do you mean it throws an exception or just ends without
running all the tests?

Thanks for the reply. it doesn't throw any exception or anything just
ends without running all the tests and its not like the loop gets
terminated at one specific point its random and could be anywhere in
the entire loop. Sometimes it runs 100 cases n will stop whereas
sometimes it'll run 150x cases n will stop there

 

[Vic]

I am not very well conversant with java and have a very basic and
limited java experience being a QA engg. I am currently looking at a
code which isn't very consistent in our testing atuomation framework.
Here is the code ->
// executes system commands and captures STDOUT and STDERR
import java.io.*;
import java.util.*;
class StreamGobbler extends Thread {
    InputStream is;
    String type;
    String result;
    StreamGobbler(InputStream is, String type){
   this.is = is;
   this.type = type;
    }
    public void run(){
   StringBuffer buffer = new StringBuffer();
   try{
       InputStreamReader isr = new InputStreamReader(is);
       BufferedReader br = new BufferedReader(isr);
       String line = null;
       while((line = br.readLine()) != null){
           buffer.append(line);
           buffer.append("\n");
       }
   }
   catch(IOException ioe){
       ioe.printStackTrace();
   }
   finally{
       result = buffer.toString();
   }
    }
    public String getResult(){
   return result;
    }
}
class NewCommandExecuter {
    public String stdout = null;
    public String stderr = null;
    private String line;
    public NewCommandExecuter (String cmd) {
   try{
       //VikP - This needed to be modified to incorporate the change in
syntax for executing getData.pl by Mark Silverman for PR#5026342
       System.out.println("New Command Executer: EXECUTING COMMAND:
"+cmd);
       String [] execmd = {"bash","-c",cmd};
       //Process proc = Runtime.getRuntime().exec(new String[]{"bash","-
c",cmd});
       Process proc = Runtime.getRuntime().exec(execmd);
       //Process proc = Runtime.getRuntime().exec(cmd);
       StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream
(),"ERROR");
       StreamGobbler outputGobbler = new StreamGobbler
(proc.getInputStream(),"OUTPUT");
       errorGobbler.start();
       outputGobbler.start();
       int val = proc.waitFor();
       System.out.println("ExitValue for Command "+cmd+" is: "+val);
       stdout = outputGobbler.getResult();
       stderr = errorGobbler.getResult();
       //VikP - Waits for these threads to die
       errorGobbler.join();
       outputGobbler.join();
       //
   }
   catch (Throwable t){
           t.printStackTrace();
   }
   if (stdout.equals(""))
       stdout = null;
   if (stderr.equals(""))
       stderr = null;
    }
}
I call the the above in a loop (of about 2000 cases) as ->
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);
// verity xml output
       if (xmlOut.stdout == null)
           throw new Exception ("Xml-out query '"+xmlOutCmd+"' didn't return
any results. STDERR returned:\n"+xmlOut.stderr);
       // save output to a temp file
       writeStringToFile(tempXmlFile, xmlOut.stdout); // save xml output
to a file
The problem that I am facing is it works fine but  it doesn't go
through the entire loop (all the test cases we have) and almost always
gets terminated prematurely completing only a partial list of cases.
So that makes me wonder if there is anything wrong in the
StreamGobbler and/or NewCommandExecuter. I tried a bunch of things in
those classes but definitely I am missing something here. Please help..
Thanks

By 'terminated', do you mean it throws an exception or just ends without
running all the tests?

Knute Johnson
email s/nospam/knute2009/

Thanks for the reply. it doesn't throw any exception or anything just
ends without running all the tests and its not like the loop gets
terminated at one specific point its random and could be anywhere in
the entire loop. Sometimes it runs 100 cases n will stop whereas
sometimes it'll run 150x cases n will stop there

I am running this automation on
Solaris 10 3/05 s10_74L2a SPARC with JDK_HOME set as /home/qa/
jdk1.5.0_02 (so I assume the jdk version in 1.5)
I don't know if it matters but thought of mentioning. Thanks

 

[Vic]

On Mar 6, 9:56 am, Knute Johnson <[email protected]>
wrote:

<snip code>




There is a small opportunity for error in that you are retrieving the stream
gobbler results before the threads have finished. That is, the test process
may have finished, but some output in the pipeline is still being processed
by the stream gobbler.  getResult will return null because the results are
not transferred until the reading loop has finished.

The answer to your problem is in the loop code (please show it) because the
call to the executer must either return or throw an exception.   What are
the loop exit conditions? What happens to any exceptions (e.g. Xml-out
query... exception) ? What did the JVM do after the loop exited?

Matt Humphreyhttp://www.iviz.com/

Let me try moving the
stdout = outputGobbler.getResult();
stderr = errorGobbler.getResult();
after the threads have finished.

I just checked and there was no try catch around
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);
so I've put that now and will try to rerun the automation and see if
what I get. I'll give an update soon. Thanks for your reply

 

[Vic]

Let me try moving the
 stdout = outputGobbler.getResult();
 stderr = errorGobbler.getResult();
after the threads have finished.

I just checked and there was no try catch around
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);
so I've put that now and will try to rerun the automation and see if
what I get. I'll give an update soon. Thanks for your reply

I just reran the automation again after moving the getResult() after
the thread join() and putting the NewCommandExecuter in a try, catch
block and I got the results for all the test cases in the loop and the
automation finished in entirety successfully. I'll keep an eye on the
behavior for the next couple of days and see if its all working and
will let you know if its working. Thank you so much for the help so
far on this topic; really appreciate it.

 

[Robert Klemme]

I am not very well conversant with java and have a very basic and
limited java experience being a QA engg. I am currently looking at a
code which isn't very consistent in our testing atuomation framework.
Here is the code ->
// executes system commands and captures STDOUT and STDERR
import java.io.*;
import java.util.*;

class StreamGobbler extends Thread {
InputStream is;
String type;
String result;

StreamGobbler(InputStream is, String type){
this.is = is;
this.type = type;
}

public void run(){
StringBuffer buffer = new StringBuffer();

try{
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String line = null;
while((line = br.readLine()) != null){
buffer.append(line);
buffer.append("\n");
}
}
catch(IOException ioe){

I'd replace that with a catch(Throwable) in order to not loose any error
conditions.

ioe.printStackTrace();
}
finally{
result = buffer.toString();
}
}

public String getResult(){
return result;
}
}

class NewCommandExecuter {
public String stdout = null;
public String stderr = null;
private String line;

public NewCommandExecuter (String cmd) {
try{
//VikP - This needed to be modified to incorporate the change in
syntax for executing getData.pl by Mark Silverman for PR#5026342
System.out.println("New Command Executer: EXECUTING COMMAND:
"+cmd);
String [] execmd = {"bash","-c",cmd};
//Process proc = Runtime.getRuntime().exec(new String[]{"bash","-
c",cmd});
Process proc = Runtime.getRuntime().exec(execmd);
//Process proc = Runtime.getRuntime().exec(cmd);
StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream
(),"ERROR");
StreamGobbler outputGobbler = new StreamGobbler
(proc.getInputStream(),"OUTPUT");

errorGobbler.start();
outputGobbler.start();

int val = proc.waitFor();
System.out.println("ExitValue for Command "+cmd+" is: "+val);

stdout = outputGobbler.getResult();
stderr = errorGobbler.getResult();

//VikP - Waits for these threads to die
errorGobbler.join();
outputGobbler.join();
//

You must first join and then fetch results. Otherwise you cannot be
sure that you got all the output properly.

}
catch (Throwable t){
t.printStackTrace();
}

if (stdout.equals(""))
stdout = null;

if (stderr.equals(""))
stderr = null;
}
}

I call the the above in a loop (of about 2000 cases) as ->
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);

// verity xml output
if (xmlOut.stdout == null)
throw new Exception ("Xml-out query '"+xmlOutCmd+"' didn't return
any results. STDERR returned:\n"+xmlOut.stderr);

// save output to a temp file
writeStringToFile(tempXmlFile, xmlOut.stdout); // save xml output
to a file

The problem that I am facing is it works fine but it doesn't go
through the entire loop (all the test cases we have) and almost always
gets terminated prematurely completing only a partial list of cases.
So that makes me wonder if there is anything wrong in the
StreamGobbler and/or NewCommandExecuter. I tried a bunch of things in
those classes but definitely I am missing something here. Please help.
Thanks

No errors? No exceptions? Are you sure it's caused by the exeternal
command execution? What if you replace that with a dummy which simply
prints command lines to stdout?

Cheers

robert

 

[Lew]

I'd replace that with a catch(Throwable) in order to not loose [sic] any error
conditions.

That's usually terrible advice.

The problem here is that the exception isn't handled, and is therefore loosed
(actual meaning) on the rest of the program.

Both of y'all need to study exception-handling strategies.

The basic tenet is "catch, log, and gracefully exit."

Not all Throwables are equally catchable, nor as easy to escape from.

 

[Robert Klemme]

I'd replace that with a catch(Throwable) in order to not loose [sic]
any error conditions.

That's usually terrible advice.

Lew, while I generally appreciate your comments I do not find this one
especially helpful as you do not provide any reasoning. I'll present my
reasoning: with a catch all on a thread's main level (method run) and
output of stack trace the code does exactly what you request (catch,
log, exit gracefully).

Having said that there are many other things that can be improved but I
did not want start rewriting the whole piece.

The problem here is that the exception isn't handled, and is therefore
loosed (actual meaning) on the rest of the program.

You got me stumped - is "loosed" proper English? Is this some form of
colloquial English that I'm missing?

The basic tenet is "catch, log, and gracefully exit."

See above.

Not all Throwables are equally catchable, nor as easy to escape from.

Well, yes. Could be that there is an Error which would prevent
subsequent output of the Exception. But at least the single catch block
for Throwable makes sure that everything that can be handled is caught
here. You could of course have separate handlers for different Thrable
subclasses but since we're in run() there's little other to do than to
output the exception and terminate. Of course, when using a proper
logging framework it would make more sense to distinguish logging levels
etc.

Cheers

robert

 

[Vic]

I am not very well conversant with java and have a very basic and
limited java experience being a QA engg. I am currently looking at a
code which isn't very consistent in our testing atuomation framework.
Here is the code ->
// executes system commands and captures STDOUT and STDERR
import java.io.*;
import java.util.*;

class StreamGobbler extends Thread {
    InputStream is;
    String type;
    String result;

    StreamGobbler(InputStream is, String type){
   this.is = is;
   this.type = type;
    }

    public void run(){
   StringBuffer buffer = new StringBuffer();

   try{
       InputStreamReader isr = new InputStreamReader(is);
       BufferedReader br = new BufferedReader(isr);
       String line = null;
       while((line = br.readLine()) != null){
           buffer.append(line);
           buffer.append("\n");
       }
   }
   catch(IOException ioe){

I'd replace that with a catch(Throwable) in order to not loose any error
conditions.

       ioe.printStackTrace();
   }
   finally{
       result = buffer.toString();
   }
    }

    public String getResult(){
   return result;
    }
}

class NewCommandExecuter {
    public String stdout = null;
    public String stderr = null;
    private String line;

    public NewCommandExecuter (String cmd) {
   try{
       //VikP - This needed to be modified to incorporate the change in
syntax for executing getData.pl by Mark Silverman for PR#5026342
       System.out.println("New Command Executer: EXECUTING COMMAND:
"+cmd);
       String [] execmd = {"bash","-c",cmd};
       //Process proc = Runtime.getRuntime().exec(new String[]{"bash","-
c",cmd});
       Process proc = Runtime.getRuntime().exec(execmd);
       //Process proc = Runtime.getRuntime().exec(cmd);
       StreamGobbler errorGobbler = new StreamGobbler(proc.getErrorStream
(),"ERROR");
       StreamGobbler outputGobbler = new StreamGobbler
(proc.getInputStream(),"OUTPUT");

       errorGobbler.start();
       outputGobbler.start();

       int val = proc.waitFor();
       System.out.println("ExitValue for Command "+cmd+" is: "+val);

       stdout = outputGobbler.getResult();
       stderr = errorGobbler.getResult();

       //VikP - Waits for these threads to die
       errorGobbler.join();
       outputGobbler.join();
       //

You must first join and then fetch results.  Otherwise you cannot be
sure that you got all the output properly.

   }
   catch (Throwable t){
           t.printStackTrace();
   }

   if (stdout.equals(""))
       stdout = null;

   if (stderr.equals(""))
       stderr = null;
    }
}

I call the the above in a loop (of about 2000 cases) as ->
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);

// verity xml output
       if (xmlOut.stdout == null)
           throw new Exception ("Xml-out query '"+xmlOutCmd+"' didn't return
any results. STDERR returned:\n"+xmlOut.stderr);

       // save output to a temp file
       writeStringToFile(tempXmlFile, xmlOut.stdout); // save xml output
to a file

The problem that I am facing is it works fine but  it doesn't go
through the entire loop (all the test cases we have) and almost always
gets terminated prematurely completing only a partial list of cases.
So that makes me wonder if there is anything wrong in the
StreamGobbler and/or NewCommandExecuter. I tried a bunch of things in
those classes but definitely I am missing something here. Please help.
Thanks

No errors?  No exceptions?  Are you sure it's caused by the exeternal
command execution?  What if you replace that with a dummy which simply
prints command lines to stdout?

Cheers

        robert

Sorry to bother you guys and I am still not able to get it running in
entirety. I have been following the automation results and its still
not running 100% (stops in the middle). I moved getResult() after the
join and used Throwable as follows -l
errorGobbler.start();
outputGobbler.start();

int val = proc.waitFor();
System.out.println("ExitValue for Command "+cmd+" is: "+val);

//VikP - Waits for these threads to die
errorGobbler.join();
outputGobbler.join();
//

stdout = outputGobbler.getResult();
stderr = errorGobbler.getResult();
}
catch (Throwable t){
t.printStackTrace();
}

Also put a Try catch with Throwable in the java file from where
NewCommandExecuter gets called as
try{
NewCommandExecuter xmlOut = new NewCommandExecuter(xmlOutCmd);

/*catch (Exception e){
System.out.println("The xmlOut command didn't get executed
properly");
this.getStackTrace(e);
}*/

// verity xml output
if (xmlOut.stdout == null)
throw new Exception ("Xml-out query '"+xmlOutCmd+"' didn't return
any results. STDERR returned:\n"+xmlOut.stderr);

// save output to a temp file
writeStringToFile(tempXmlFile, xmlOut.stdout); // save xml output
to a file
}
catch (Throwable e){
System.out.println("Error executing xmlOutCmd");
e.printStackTrace();
}

To give you guys an idea about the way we run automation is
we do jloader -v <version> >&! out/<log file>
So jloader basically calls LoaderAutomationRun.java (starts the loop)
and calls LoaderTestcase.java in the loop for each test case and
inside LoaderTestcase.java it runs the commands using
NewCommandExecuter.

I just looked a thte out/<log file> and it just shows something like
this ->
TESTCASE Transaction836
Start generation all loader files
DEBUG: validating(transaction/*
DEBUG: NEW load order transaction/1,
DEBUG: FINAL NEW load order transaction/1,
Executing generateLoaderFile(transaction/1, 0)
dTrans: null
Executing generateLoaderFile, adding Transaction836.0.lrd
LOADER FILES CREATED 1
DEBUG: Created 1 loader files
New Command Executer: EXECUTING COMMAND: /home/qauser/620currentLOADER/
bin/loader -u genevaman -w qa -f Transaction836.0.lrd

As you can see the last line, the New Command Executer never returned.
Sorry to bother you guys but I really want to get this solved soon so
any help on this would be appreciated. Thanks in advance

 

[Vic]

Ok, so what you're observing is that diagnostic line "ExitValue for Command"
does not appear in the output, but you still haven't shown the loop code or
said what you mean by "never returned"  Now that we understand the previous
problem, we know that if the test returns null the loop will exit and your
main program will still stop without any messages.  This new case, however,
looks like the test process itself never finishes and the test driver does
not stop.  Is that the case?  You need to say more about what that loop is
and does.

If your test driver is hanging, I'd guess that the test process itself is
faulty.  Perhaps it's requesting input or just plain stuck.  If your test
driver is halting, the answer will be in how it halts.  It may fail if New
Command Executer throws an exception prior to waitFor.  It's not clear that
stderr exceptions are appearing in your output and we know that the null
will cause your main to halt with no messages.

Lew also mentioned one possible case by which the StreamGobbler's assignment
of the result field in its own thread is not visible to the driver thread
because of the lack of synchronization.  Put the keyword "volatile" on the
String result declaration in StreamGobbler to fix this.

Matt Humphreyhttp://www.iviz.com/

Thanks for the reply as I mentioned earlier jloader.java calls ->
LoaderAutomationRun.java

constructor -> public jloader(Vector vTestcasesList, int count) {
LoaderAutomationRun myLoaderAutomationRun = new
LoaderAutomationRun("Loader",count);
myLoaderAutomationRun.run (vTestcasesList,count);
System.exit(0);
}
public static void main (String args[]) {
Automation setupAutomation = new Automation();
jloader myAutomation;
parseCommandLineArguments (args, "loader");


else {
//0 is specified for the Standard Loader
myAutomation = new jloader(vTestcasesList, 0);
}
}

and the loop inside LoaderAutomationRun.java which calls
LoaderTestcase.java and looks like this ->
for (int i = 0; i < vTestcasesList.size(); i++) {

sTestcaseName = vTestcasesList.elementAt(i).toString();
iExpected++;
iRan++;
System.out.println("\nTESTCASE "+sTestcaseName);

//VicP- copying the testcase name
LoaderTestcase.sTestcaseNameTemp = sTestcaseName;
//

//VicP - Transaction324 is working now to commenting out
if(sTestcaseName.equals("Transaction324"))
continue;

try {
myTC = new LoaderTestcase(sTestcaseName);
myTC.run();
bPassed = true;
} catch (Exception e) {
iDiffs++;
String exceptionFileName = results_log_dir + sTestcaseName +
".exception";
// save exception to log file
try {
Automation.writeStringToFile (exceptionFileName,
Automation.getStackTrace(e));
System.err.println ("\n\nTESTCASE "+sTestcaseName+" FAILED
\n"+Automation.getStackTrace(e)+"Exception saved to file
'"+exceptionFileName+"'\n*** TESTCASE END\n");
} catch (Exception ex) {
System.err.println("Exception occured while trying to write
to file '"+exceptionFileName+"'\n"+Automation.getStackTrace(ex));
}
bPassed = false;
}
System.err.println("EXP:" +iExpected+" RAN:"+iRan+"
DIFF:"+iDiffs);
System.out.println("TESTCASE "+sTestcaseName
+"\n----------------------");
if (bPassed)
System.err.println("->PASSED");
else
System.err.println("->FAILED");
System.err.println("\n\n\n\n\n\n\n");
}

myTC.run() funcation in LoaderTestcase.java above calls
NewCommandExecuter.java to execute the command
what I meant by never returns is that the automation stops/gets killed
either in jloader.java, LoaderAutomationRun.java or while executing
NewcommandExecuter (not sure which one is causing the automation to
quit in the middle of the run). As you mentioned there could be
something wrong with jloader.java which gets killed somehow
thanks in advance

 

[Vic]

Ok, so what you're observing is that diagnostic line "ExitValue for Command"
does not appear in the output, but you still haven't shown the loop code or
said what you mean by "never returned"  Now that we understand the previous
problem, we know that if the test returns null the loop will exit and your
main program will still stop without any messages.  This new case, however,
looks like the test process itself never finishes and the test driver does
not stop.  Is that the case?  You need to say more about what that loop is
and does.

If your test driver is hanging, I'd guess that the test process itself is
faulty.  Perhaps it's requesting input or just plain stuck.  If your test
driver is halting, the answer will be in how it halts.  It may fail if New
Command Executer throws an exception prior to waitFor.  It's not clear that
stderr exceptions are appearing in your output and we know that the null
will cause your main to halt with no messages.

Lew also mentioned one possible case by which the StreamGobbler's assignment
of the result field in its own thread is not visible to the driver thread
because of the lack of synchronization.  Put the keyword "volatile" on the
String result declaration in StreamGobbler to fix this.

Matt Humphreyhttp://www.iviz.com/

Thanks for the reply as I mentioned earlier jloader.java calls ->
LoaderAutomationRun.java

constructor -> public jloader(Vector vTestcasesList, int count) {
        LoaderAutomationRun myLoaderAutomationRun = new
LoaderAutomationRun("Loader",count);
        myLoaderAutomationRun.run (vTestcasesList,count);
        System.exit(0);
    }
public static void main (String args[]) {
       Automation setupAutomation = new Automation();
       jloader myAutomation;
       parseCommandLineArguments (args, "loader");

      else {
           //0 is specified for the Standard Loader
           myAutomation = new jloader(vTestcasesList, 0);
       }
    }

and the loop inside LoaderAutomationRun.java which calls
LoaderTestcase.java and looks like this ->
for (int i = 0; i < vTestcasesList.size(); i++) {

            sTestcaseName = vTestcasesList.elementAt(i).toString();
            iExpected++;
            iRan++;
            System.out.println("\nTESTCASE "+sTestcaseName);

            //VicP- copying the testcase name
            LoaderTestcase.sTestcaseNameTemp = sTestcaseName;
            //

            //VicP - Transaction324 is working now to commenting out
            if(sTestcaseName.equals("Transaction324"))
                continue;

            try {
                myTC = new LoaderTestcase(sTestcaseName);
                myTC.run();
                bPassed = true;
            } catch (Exception e) {
                iDiffs++;
                String exceptionFileName = results_log_dir + sTestcaseName +
".exception";
                // save exception to log file
                try {
                    Automation.writeStringToFile (exceptionFileName,
Automation.getStackTrace(e));
                    System.err.println ("\n\nTESTCASE "+sTestcaseName+" FAILED
\n"+Automation.getStackTrace(e)+"Exception saved to file
'"+exceptionFileName+"'\n*** TESTCASE END\n");
                } catch (Exception ex) {
                    System.err.println("Exception occured while trying to write
to file '"+exceptionFileName+"'\n"+Automation.getStackTrace(ex));
                    }
                bPassed = false;
            }
            System.err.println("EXP:" +iExpected+" RAN:"+iRan+"
DIFF:"+iDiffs);
            System.out.println("TESTCASE "+sTestcaseName
+"\n----------------------");
            if (bPassed)
                System.err.println("->PASSED");
            else
                System.err.println("->FAILED");
            System.err.println("\n\n\n\n\n\n\n");
        }

myTC.run() funcation in LoaderTestcase.java above calls
NewCommandExecuter.java to execute the command
what I meant by never returns is that the automation stops/gets killed
either in jloader.java, LoaderAutomationRun.java or while executing
NewcommandExecuter (not sure which one is causing the automation to
quit in the middle of the run). As you mentioned there could be
something wrong with jloader.java which gets killed somehow
thanks in advance

o forgot to mention, myTC.run() looks like this ->
public void run () throws java.io.IOException, Exception {
try {
LoaderFile[] aLoaderFilesObjects = generateAllLoaderFiles();
logDebugMsg("Created "+aLoaderFilesObjects.length+" loader
files", 1);
if (aLoaderFilesObjects.length == 0)
throw new Exception("No loader files have been created");

for (int j = 0; j < aLoaderFilesObjects.length; j++) {
FileLoader currentFileLoader = new FileLoader (aLoaderFilesObjects
[j], (j==aLoaderFilesObjects.length-1));
if ( j == (aLoaderFilesObjects.length-1) ){
if((sTestcaseType.intern().equals("Positive") ||
sTestcaseType.intern().equals("Negative")) &&
LoaderAutomationRun.counter == 0)
verify(currentFileLoader);
}
//loadCleanUpRecords ();
}
} catch (Exception e) {
//loadCleanUpRecords ();
throw e;
}
}

 

[Vic]

Thanks for the reply as I mentioned earlier jloader.java calls ->
LoaderAutomationRun.java

constructor -> public jloader(Vector vTestcasesList, int count) {
        LoaderAutomationRun myLoaderAutomationRun = new
LoaderAutomationRun("Loader",count);
        myLoaderAutomationRun.run (vTestcasesList,count);
        System.exit(0);
    }
public static void main (String args[]) {
       Automation setupAutomation = new Automation();
       jloader myAutomation;
       parseCommandLineArguments (args, "loader");

      else {
           //0 is specified for the Standard Loader
           myAutomation = new jloader(vTestcasesList, 0);
       }
    }

and the loop inside LoaderAutomationRun.java which calls
LoaderTestcase.java and looks like this ->
for (int i = 0; i < vTestcasesList.size(); i++) {

            sTestcaseName = vTestcasesList.elementAt(i).toString();
            iExpected++;
            iRan++;
            System.out.println("\nTESTCASE "+sTestcaseName);

            //VicP- copying the testcase name
            LoaderTestcase.sTestcaseNameTemp = sTestcaseName;
            //

            //VicP - Transaction324 is working now to commenting out
            if(sTestcaseName.equals("Transaction324"))
                continue;

            try {
                myTC = new LoaderTestcase(sTestcaseName);
                myTC.run();
                bPassed = true;
            } catch (Exception e) {
                iDiffs++;
                String exceptionFileName = results_log_dir + sTestcaseName +
".exception";
                // save exception to log file
                try {
                    Automation.writeStringToFile (exceptionFileName,
Automation.getStackTrace(e));
                    System.err.println ("\n\nTESTCASE "+sTestcaseName+" FAILED
\n"+Automation.getStackTrace(e)+"Exception saved to file
'"+exceptionFileName+"'\n*** TESTCASE END\n");
                } catch (Exception ex) {
                    System.err.println("Exception occured while trying to write
to file '"+exceptionFileName+"'\n"+Automation.getStackTrace(ex));
                    }
                bPassed = false;
            }
            System.err.println("EXP:" +iExpected+" RAN:"+iRan+"
DIFF:"+iDiffs);
            System.out.println("TESTCASE "+sTestcaseName
+"\n----------------------");
            if (bPassed)
                System.err.println("->PASSED");
            else
                System.err.println("->FAILED");
            System.err.println("\n\n\n\n\n\n\n");
        }

myTC.run() funcation in LoaderTestcase.java above calls
NewCommandExecuter.java to execute the command
what I meant by never returns is that the automation stops/gets killed
either in jloader.java, LoaderAutomationRun.java or while executing
NewcommandExecuter (not sure which one is causing the automation to
quit in the middle of the run). As you mentioned there could be
something wrong with jloader.java which gets killed somehow
thanks in advance

o forgot to mention, myTC.run() looks like this ->
public void run () throws java.io.IOException, Exception {
        try {
            LoaderFile[] aLoaderFilesObjects = generateAllLoaderFiles();
            logDebugMsg("Created "+aLoaderFilesObjects.length+" loader
files", 1);
            if (aLoaderFilesObjects.length == 0)
                throw new Exception("No loader files have been created");

            for (int j = 0; j < aLoaderFilesObjects.length; j++) {
                FileLoader currentFileLoader = new FileLoader (aLoaderFilesObjects
[j], (j==aLoaderFilesObjects.length-1));
                if ( j == (aLoaderFilesObjects.length-1) ){
                    if((sTestcaseType.intern().equals("Positive") ||
sTestcaseType.intern().equals("Negative")) &&
LoaderAutomationRun.counter == 0)
                        verify(currentFileLoader);
                }
                //loadCleanUpRecords ();
            }
        } catch (Exception e) {
            //loadCleanUpRecords ();
            throw e;
        }
    }

I think the NewCommandExecuter is all good as mentioned by someone
here earlier and the problem is I think in the program where it gets
called. I had sent some code snippets earlier. Any idea how to make it
better? Also I was wondering many of our methods that calls
NewCommandExecuter specify throws java.io.Exception, Exception and
then also has a try catch block around the code where
NewCommandExecuter.java gets executed and then there is catch
(Thrwoable t) in the NewCommandExecuter.java as well. Is it doing too
much of stuff with the exception handling and inspite of all this the
program is getting terminated in the middle which is probably because
the exception handling isn't correctly written or something. I am
going to try to rework some of the stuff and this code has been around
for a long time and nobody really looked at it till now.