Maximum length of Command Line Argument

[balamurugan.murthy]

I am trying to call a java class along with a command line argument from a shell script . The line of code in the shell script will be like as follows:


arg_01=`echo $file_list | sed 's/\s/,/g'`
touch /home/ora/crontest.txt
java -classpath $webapp_classpath com.example.it.util.InvokeBatch $arg_01
#End of shell Script


From java documentation I understand that the maximum command line argument length can be up to a maximum of Integer.MAX_VALUE (approximately 2147483647).

But does this hold good when i am calling this java class from within a shell script too ? I believe that there is a maximum length applicable to command line arguments in UNIX. In any way does this value restricts the Integer.MAX_VALUE that JAVA allows?

Thanks in Advance for your help.

[Norm]

Are you having a java problem?
Does your command line work?
If so write a test program with a loop that increases the commandline length each iteration until something bad happens. That could give you a feel for what the limits are.

[toadaly]

Unix shells limit the maximum command line. There's nothing java can do about that since the restriction happens before the jvm is even invoked. In this instance, there is no need to pass in arg_01 since it's an environment variable.

You can read it in programmatically using

String arg_01 = System.getenv("arg_01");

[gcalvin]

This doesn't look like a great approach to me at all, but let me be sure I'm understanding it correctly.

1. You are piping whatever is in $file_list through sed, and replacing all white space (including newlines) with commas, forming one really big string.
2. You intend to pass that really big string as a single arg to a Java program, which will presumably parse the arg, using the comma as a delimiter.
3. You're concerned that $file_list, and therefore the string you generate, could get VERY big, and you are concerned about limits.

So why not simply pipe $file_list directly to your Java program, and read a line at a time from STDIN?

        public static void main(String[] args) {
                Scanner sc = new Scanner(System.in);
                while (sc.hasNextLine()) {
                        String line = sc.nextLine();
                        processLine(line); // write this method
                }
        }

-Gary-

[balamurugan.murthy]

Gary and toadaly,

Thank you very much for your suggestions.

Gary,
Your understanding is correct and Yes i felt that what i was trying to do was a bad approach but could not think of an alternate approach until i found your reply. But to be frank, I am still not able to get your solution properly. Do you say that the parameter passed while creating the scanner object must be the String ($file_list) that I am trying to pass via command line? and I should get that value from the environment using the method that toadaly has mentioned ? (System.getenv("arg_01")).

Am i getting your points here correctly. Please correct if my understanding is wrong.

[balamurugan.murthy]

Alternately I was planning to write the variable $file_list contents into a single file and pass the file name as a parameter to the java class. Is this an efficient approach ? Please advice.

- Bala

[gcalvin]

Gary and toadaly,

Thank you very much for your suggestions.

Gary,
Your understanding is correct and Yes i felt that what i was trying to do was a bad approach but could not think of an alternate approach until i found your reply. But to be frank, I am still not able to get your solution properly. Do you say that the parameter passed while creating the scanner object must be the String ($file_list) that I am trying to pass via command line? and I should get that value from the environment using the method that toadaly has mentioned ? (System.getenv("arg_01")).

Am i getting your points here correctly. Please correct if my understanding is wrong.

No, not a parameter, but a pipe. Just as you are doing

echo $file_list | sed 's/\s/,/g'

you can do instead

echo $file_list | java MyClass

-Gary-

[balamurugan.murthy]

Thanks for the clarification, Gary.

[balamurugan.murthy]

Gary, I am able to work out your successfully but got another question here:

Does a UNIX variable ($file_list that i am using here) has any limitations ? In other words, what can be the maximum size of a string variable in UNIX ? Is it OS dependent or Shell dependent?

[gcalvin]

Gary, I am able to work out your successfully but got another question here:

Does a UNIX variable ($file_list that i am using here) has any limitations ? In other words, what can be the maximum size of a string variable in UNIX ? Is it OS dependent or Shell dependent?

We're getting away from Java here, and therefore out of scope for this forum. I don't know the direct answer to your question, but in my experience, when I'm concerned with running up against limits like these, it's due to a poor architectural choice somewhere up the line. In your case, it's probably simple -- where does $file_list come from? Instead of storing anything in a $file_list variable, can you simply have that operation pipe directly to your Java program? I can tell you that there is no practical limit to pipes, and that something like this is quite common:

# copies a directory, or even an entire filesystem to another host,
# preserving file ownership and permissions -- this could be terabytes of
# data
(cd /source/file/path && tar cpzf - . ) | (ssh targethost "cd /target/file/path && tar xvzf -")

-Gary-

[gcalvin]

For a more concrete example of what I'm talking about, type this into a file PipeExample.java:

import java.util.Scanner;

public class PipeExample {
	private static int lineNumber = 0;

	private static void processLine(String line) {
		// I hope it's obvious that this method can do
		// whatever you want it to do. For this example,
		// we simply print the data to the console, preceded
		// by a line number
		System.out.println(++lineNumber + ") " + line);
		
	}

	public static void main(String[] args) {
		// Demonstrate receiving data from a pipe and processing it
		Scanner sc = new Scanner(System.in);
		while (sc.hasNextLine()) {
			// get your data, one line at a time from the pipe (stdin)
			String line = sc.nextLine();
			// pass the data to a helper method for processing
			processLine(line);
		}

	}

}

Compile it.

hostname:~ username$ [B]javac PipeExample.java[/B]

Then run it, passing a directory of every file on your system. (You can Ctrl-C to break out of the run.)

hostname:~ username$ [B]ls -laR / | java PipeExample[/B]

-Gary-

[balamurugan.murthy]

Thanks Gary. your reply cleared my confusion and I got a better solution now for my problem. I applied it to my problem and its working fine without any issues. Thanks a million again.