Slow Output reading with Runtime.getRuntime().exec()

[gmcouto]

Hello, it is my first post here. I really need your help guys.

I am developing a GUI for a C program. Im using Java to execute a command with Runtime.getRuntime.exec().... and using a different thread to monitor the InputStream.

The program generetas several lines of output per second.
I need to read the program output and show the progress on the GUI.

The problem is that it takes a lot of seconds to start printing something... then it prints everything that it should have printed before... and stops printing for a lot more seconds(like 10+ seconds)... then it prints everything again just once.... and stops.... and so on until the program finishes.

public class CommandLineRun {
    Process p;
    Runtime rt;
    BufferedReader stdInput, stdError;
    private boolean running;

    public CommandLineRun(File f,String[] args){
        rt = Runtime.getRuntime();
        try {
            p = rt.exec(f.getCanonicalPath(), args, f.getCanonicalFile().getParentFile());

            running=true;
        } catch (IOException ex) {
            Logger.getLogger(CommandLineRun.class.getName()).log(Level.SEVERE, null, ex);
        }
        stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
        stdError = new BufferedReader(new InputStreamReader(p.getErrorStream()));
    }
    public BufferedReader getOutput(){
        return stdInput;
    }
    public BufferedReader getError(){
        return stdError;
    }
    public void halt(){
        p.destroy();
        running=false;
    }
    public boolean procDone() {
        try {
            int v = p.exitValue();
            running=false;
            return true;
        }
        catch(IllegalThreadStateException e) {
            return false;
        }
    }
    public boolean isRunning() {
        return running;
    }
}

public class monExatoModuleOut extends Thread {

    CommandLineRun run;

    public monExatoModuleOut(CommandLineRun run) {
        this.run = run;
    }

    public void run() {
        String s;
        while (!run.procDone() && run.isRunning()) {
            try {
                while (( s = run.getError().readLine()) != null) {
                    System.out.println(s);
                }
            } catch (IOException ex) {
                Logger.getLogger(execExatoModuleThread.class.getName()).log(Level.SEVERE, null, ex);
            }
        }

    }
}

The fact is the program I need to execute is CPU intensive, so I'm afraid that the thread that puts info on InputStream barely gets CPU.

Is there anything to do to fix this and print the output something about real time?

---
Sorry for my poor english

[Fubarable]

The key may be in your GUI code. Is this a Swing program? If so, are you running your long-running process in a different thread than the GUI's main thread (the EDT)? If I were doing this with Java 1.6, I'd consider using a SwingWorker object to be sure that the C program is run on a background thread, and then would display the text via the SwingWorker's publish/process method pair.

[gmcouto]

Runtime.getRuntime.exec() starts a new process, so it is definitely in a different thread(am I wrong?).

As you can see, even the class that monitors output runs in a different thread.

---

I'll try to do something with SwingWorker. brb.

Thanks for your help by the way

[Fubarable]

Runtime.getRuntime.exec() starts a new process, so it is definitely in a different thread(am I wrong?).

Sorry, but I think this statement above is wrong. The new process is created in the thread that spawns it, and can tie up that thread.

As you can see, even the class that monitors output runs in a different thread.
---
I'll try to do something with SwingWorker. brb.
Thanks for your help by the way

You're welcome. If it fails, then you may with to try to create a very simple and small program that demonstrates your problem.

Best of luck.

[gmcouto]

Sorry, but I couldn't figure out a way to do it using SwingWorkers.
I tought that different processes couldn't share the same thread.

Ive just made a debug version of my "console reader" and it might be easier to understand what I want

public class CmdOutputPrinter {

    Process p;
    Runtime rt;
    BufferedReader stdInput;
    private String s;

    public CmdOutputPrinter() {
        rt = Runtime.getRuntime();
        try {
            p = rt.exec("./exato x corpus.txt");
        } catch (IOException ex) {
        }
        stdInput = new BufferedReader(new InputStreamReader(p.getInputStream()));
        while (!procDone()) {
            try {
                while ((s = stdInput.readLine()) != null) {// it hangs here, waiting to receive readLine()
                    //everything else runs fine
                    System.out.println(s);
                }
            } catch (IOException ex) {
                Logger.getLogger(CmdRunWorker.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
    }

    public boolean procDone() {
        try {
            int v = p.exitValue();
            return true;
        } catch (IllegalThreadStateException e) {
            return false;
        }
    }

    public static void main(String[] args) {
        new CmdOutputPrinter();
    }
}

So, when it comes to that point it hang... it stays there until every output is over... or at least, a lot of it(and hang again immediately again).

So it's like you create a GUI to a program that count seconds, and print'em every second. And the problem is that the GUI just receive the results of the original program output every ten seconds.

[Fubarable]

It sounds like, smells like, and looks like a Swing thread issue. The key is how you call this in the GUI, and you may wish to show us this. I still think that a SwingWorker would work best to solve this problem. Consider creating a very small compilable program (a GUI program) that mimics this problem, and posting it here. The smaller the better (less work and easier for us).

Much luck!

[gmcouto]

If you look at the dummy app I made last post... It only prints the results on the console.
There is no Swing in that test, and the problem remains...


Even if I create a new thread to call the runtime.exec() + a thread to monitor the stream the problem persists.... So my conclusion it is not the Swing.... (as in both of cases, with/without Swing, it hangs with BufferedReader.getLine)


And I tried to reproduce a problem compiling a sample of a C++ program... but then it works normally.
It only happens with the program I'm trying to create a GUI.(maybe because it really uses 100% of CPU)

---

This is the class that executes the program in my GUI project, and changes the values in the progress bar of my Swing panel.

public class execExatoModuleThread extends Thread {

    JTextArea taOut;
    JProgressBar pbRead;
    JProgressBar pbWrite;
    int qtdFiles;
    int qtdGrams;
    CommandLineRun run;

    public execExatoModuleThread(JTextArea ta, JProgressBar pbRead, JProgressBar pbWrite, int numFiles, int numGrams) {
        super();
        taOut = ta;
        this.pbRead = pbRead;
        this.pbWrite = pbWrite;
        qtdFiles = numFiles;
        qtdGrams = numGrams;
    }

    public void halt() {
        run.halt();
    }

    public void run() {
        taOut.setText("");
        String line;
        //mod style of bar
        pbRead.setIndeterminate(true);
        pbWrite.setIndeterminate(true);
        //change text
        pbRead.setString("Start process");
        pbWrite.setString("Start process");
        //mod style bar
        pbRead.setIndeterminate(false);
        pbWrite.setIndeterminate(false);

        System.out.println("\n\nPreparing to execute ExATO...\n");
        String[] xpar = {"x", "corpus.txt"};
        run = new CommandLineRun(Utils.ExATO, xpar);


        int iFiles = 0;
        int iGrams = 0;
        String lineout;

        while (!run.procDone() && run.isRunning()) {
            try {
                while ((lineout = run.getOutput().readLine()) != null) {

                    taOut.append(lineout + "\n");
                    System.out.println(lineout);//debug

                    if (iFiles < qtdFiles && lineout.contains("reading file")) {
                        iFiles++;
                        int valuePBread = (100 * iFiles) / qtdFiles;
                        pbRead.setValue(valuePBread);
                        pbRead.setString(iFiles + "/" + qtdFiles);
                        System.out.println(iFiles + " of " + qtdFiles + " done!");
                    } else if (iFiles == qtdFiles) {
                        pbRead.setString("Complete Read");
                        pbRead.setValue(100);
                        if (iGrams < qtdGrams && lineout.contains("generating")) {
                            iGrams++;
                            int valuePBwrite = (100 * iGrams) / (qtdGrams);
                            pbWrite.setValue(valuePBwrite);
                            pbWrite.setString(iGrams + "/" + qtdGrams);
                        } else if (iFiles == qtdFiles) {
                            pbWrite.setString("Complete Extract");
                            pbWrite.setValue(100);
                        }
                    }

                    System.out.println("Working!");
                }
            } catch (IOException ex) {
                Logger.getLogger(execExatoModuleThread.class.getName()).log(Level.SEVERE, null, ex);
            }
        }
        System.out.println("Done!");
    }
}

It is important to notice that it is a Thread.

This is the piece of code of the GUI that calls the Thread:

 private void doProcess() {
        jButton2.setEnabled(false);
        jButton3.setEnabled(true);

        String out = parameters.toString();
        File f = new File("corpus.txt");
        if (f.exists()) {
            f.delete();
        }
        try {
            f.createNewFile();
        } catch (IOException ex) {
            Logger.getLogger(ExATO_AnalysisPanel.class.getName()).log(Level.SEVERE, null, ex);
        }
        Utils.writeTxt(out, f);//generates parameters file
        System.out.println(out);//debug

        eeModule = new execExatoModuleThread(jTextArea1,jProgressBar1,jProgressBar2,parameters.getFilesLength(),parameters.getGramsQuota());
        eeModule.start(); //now the real thing happens
    }

I even tried to use a different(than the one that called Runtime.exec) thread to monitor the InputStream, but the result was the same.

I really don't know what to do.

[gmcouto]

I think I found the solution.

Java InputStream keeps reading until the buffer is flushed or filled up. That's why it hangs on ReadLine().... because the it is filling the buffer, to then send it to InputStream.

I could reproduce the problem with this little C++ program:

#include <iostream>
int main(){
	long a = 0;
	long b = 0;
	long c = 1;
	while(true){
		std::cout << "Oi " << c << "\n";//it doesnt flush, it fucks up
		while(a<1000000){ a++; }
		a=0;
		while(b<1000000){ b++; }
		b=0;
		c++;
	}
}

The only thing I did important is use "\n" for linebreak instead of endl.

So I've researched a little further and I found that endl flushes a buffer.
How could I be sure? I tested this code too:

#include <iostream>
int main(){
	long a = 0;
	long b = 0;
	long c = 1;
	while(true){
		std::cout<< "Oi " << c << "\n" << std::flush;// it flushes now, its ok
		while(a<1000000){ a++; }
		a=0;
		while(b<1000000){ b++; }
		b=0;
		c++;
	}
}

Conclusion, I need to ask the developer to change his app in need to develop a GUI that does the expected behavior.

---

Sorry for the double posting

[JosAH]

Have a look at the other BufferedReader constructor; it allows for a user defined buffer size; try a value of 80 or so. If it works you (they?) don't have to fiddle-diddle with the C++ application.

kind regards,

Jos