Results 1 to 8 of 8
  1. #1
    Looserette is offline Member
    Join Date
    Feb 2011
    Posts
    12
    Rep Power
    0

    Default Launching perl script from Java thread

    Hello all,

    sorry for asking another perl related question (I've been searching this forum before asking, and it has already got many perl related questions... but not this particular one):


    I'm running a basic java code, which is receiving input from 1 socket (will be several sockets if I manage to get an answer :)).
    for each new input on this socket, I'm starting a new thread, because I need some processing to do on this input.. while other inputs are coming in, and they can't wait for the 1st input to finish.

    anyway, for each input, I need to run this input through a perl script, which outputs a specific formatted string.

    I'm using Runtime.getRuntime().exec(cmd);

    as in:
    Java Code:
    String[] cmd = {"/usr/bin/perl","path_to_my_perl_script.pl",input};
    Process p = Runtime.getRuntime().exec(cmd);
    BufferedReader output = new BufferedReader (new InputStreamReader(p.getInputStream()));
    line = output.readLine();
    output.close();
    this is working fine...except java seems to be waiting for the perl script to end before running another one !


    => is there a way to bypass this limitation ? (I don't even know what is this limitation, since the system can handle many of this perl script instances in parallel - I can confirm that)



    I'm not sure if all that makes sense - don't hesitate to ask questions



    ps: in case you need it, here is the result on my perl -V command:
    pps: the solution "try recompiling with multiplicity ON" is really something I would like to avoid, since I cannot recompile perl ...

    bash-3.00# perl -V
    Summary of my perl5 (revision 5 version 8 subversion 4) configuration:
    Platform:
    osname=solaris, osvers=2.10, archname=sun4-solaris-64int
    uname='sunos localhost 5.10 sun4u sparc SUNW,Ultra-2'
    config_args=''
    hint=recommended, useposix=true, d_sigaction=define
    usethreads=undef use5005threads=undef useithreads=undef usemultiplicity=undef
    useperlio=define d_sfio=undef uselargefiles=define usesocks=undef
    use64bitint=define use64bitall=undef uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
    Compiler:
    cc='cc', ccflags ='-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -xarch=v8 -D_TS_ERRNO',
    optimize='-xO3 -xspace -xildoff',
    cppflags=''
    ccversion='Sun WorkShop', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=87654321
    d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16
    ivtype='long long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
    alignbytes=8, prototype=define
    Linker and Libraries:
    ld='cc', ldflags =''
    libpth=/lib /usr/lib /usr/ccs/lib
    libs=-lsocket -lnsl -ldl -lm -lc
    perllibs=-lsocket -lnsl -ldl -lm -lc
    libc=/lib/libc.so, so=so, useshrplib=true, libperl=libperl.so
    gnulibc_version=''
    Dynamic Linking:
    dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-R /usr/perl5/5.8.4/lib/sun4-solaris-64int/CORE'
    cccdlflags='-KPIC', lddlflags='-G'

  2. #2
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    Java has no problem exec'ing multiple processes simultaneously.

    Whatever limitation you're running into is almost assuredly a bug in either your java code or your perl script.

    What happens when you try to exec another process?

  3. #3
    Looserette is offline Member
    Join Date
    Feb 2011
    Posts
    12
    Rep Power
    0

    Default

    Quote Originally Posted by toadaly View Post
    What happens when you try to exec another process?
    Good question - I should have said it in my original post:

    - I tried replacing the perl script command by :
    String[] cmd = {"/usr/bin/ls","-al","/tmp"};
    => I get (I think) the same result: java waits for the first ls to complete before running the next one (although it's hard to check, since ls is a bit too fast ... I'll try with another command that takes a bit longer)

    - I tried replacing this by:
    String[] cmd = {"/usr/bin/sleep","1"};
    => it launches max 4 sleep command at any one time (although it should be launching many more than that)

    - I tried replacing the perl script by:
    Thread.currentThread().sleep(2000);
    (sleep period of 2sec, which is at least 10 times longer than the time it takes to run my perl script)
    => it then runs fine, in parallel


    note: the "runs fine"/"has same issue" I notice is mainly because if the execution is not in parallel, the java process reaches 3000LWPS, and crashes... if the execution is in parallel, the java process can cope with that and reduces the number of LWPS quickly (even with a 2sec delay, it manages to stay under the 3000 LWPS).
    ie:
    - with the /usr/bin/sleep 1 command, the java process reaches 3000 and crashes
    - with the Thread.currentThread().sleep(2000); , the java process never reaches 3000 and is able to cope with the load.



    Thanks for the suggestion !

    if you have any other idea/tests, let me know, because I'm stuck here :(
    Last edited by Looserette; 02-23-2011 at 02:14 PM.

  4. #4
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    Try this:

    - create a script that does nothing but sleep, for say 60 seconds. Let's say the script is called "sleepy.pl"

    - In your code, try something like this:

    Java Code:
    long startTime = System.currentTimeMillis();
    Process p1 = Runtime.getRuntime().exec("sleepy.pl");
    Process p2 = Runtime.getRuntime().exec("sleepy.pl");
    long stopTime = System.currentTimeMillis();
    
    System.out.println((stopTime-startTime));
    What this will do, is demonstrate to you that java is not waiting for one process to finish before starting another.

    In the OP, you are reading from the stdout of the subprocess. So sure, in that case reads on the input stream are going to block until the stream ends (which doesn't happen until the subprocess exits). This isn't a java issue. The same thing would happen if you were trying to read that pipe from native code. That's how it's *supposed* to work. If you want to read the stdout of a subprocess and doing something else simultaneously, you have to use multiple threads.

  5. #5
    Looserette is offline Member
    Join Date
    Feb 2011
    Posts
    12
    Rep Power
    0

    Default

    Thanks again for your suggestions ! that's really appreciated : )


    For the stdout blocking: actually, I am already in a thread, so the blocking is intended.
    Worse than that, I actually have a:
    int exit = p.waitFor();

    since I'm already in a thread, that should - in theory - work fine : )



    Here are 2 parts of code to try to be a bit clearer: (this code is executed in a thread)

    That does work fine:
    Java Code:
    long startTime = System.currentTimeMillis();
    System.out.println("Starting command: "+counter+" at: "+startTime);
    Thread.currentThread().sleep(1000);//sleep
    long stopTime = System.currentTimeMillis();
    long diff = stopTime - startTime;
    System.out.println("Command finished: "+counter+" at: "+stopTime+" (time diff: ) "+diff);
    It results in things like:
    Starting command: 3391 at: 1298550051582
    Starting command: 3392 at: 1298550051582
    Command finished: 3388 at: 1298550052363 (time diff: ) 1000
    Command finished: 3389 at: 1298550052499 (time diff: ) 1000
    Command finished: 3390 at: 1298550052503 (time diff: ) 1000
    Command finished: 3391 at: 1298550052582 (time diff: ) 1000



    Now, with this code:
    Java Code:
    long startTime = System.currentTimeMillis();
    System.out.println("Starting command: "+counter+" at: "+startTime);
    String[] cmd = {"/usr/bin/sleep","1"};
    Process p = Runtime.getRuntime().exec(cmd);
    int exit = p.waitFor();
    long stopTime = System.currentTimeMillis();
    long diff = stopTime - startTime;
    System.out.println("Command finished: "+counter+" at: "+stopTime+" (time diff: ) "+diff);

    Both codes should be equivalent (I think), but the results are not the same:
    when the input is coming slowly, I get :
    Starting command: 1 at: 1298550634183
    Starting command: 2 at: 1298550634184
    Starting command: 3 at: 1298550634185
    Starting command: 4 at: 1298550634185
    Starting command: 5 at: 1298550634186
    Starting command: 6 at: 1298550634186
    Starting command: 7 at: 1298550634232
    Starting command: 8 at: 1298550634237
    Starting command: 15 at: 1298550634260
    Starting command: 17 at: 1298550634261
    Starting command: 18 at: 1298550634262
    Starting command: 10 at: 1298550634261
    Starting command: 13 at: 1298550634262
    Starting command: 11 at: 1298550634263
    Starting command: 12 at: 1298550634261
    Starting command: 14 at: 1298550634261
    Starting command: 16 at: 1298550634261
    Starting command: 19 at: 1298550634281
    Starting command: 20 at: 1298550634287
    Starting command: 21 at: 1298550634287
    Starting command: 22 at: 1298550634287
    Starting command: 9 at: 1298550634288
    Starting command: 23 at: 1298550634289
    Starting command: 24 at: 1298550634306
    Starting command: 25 at: 1298550634307
    Starting command: 26 at: 1298550634308
    Starting command: 27 at: 1298550634308
    Starting command: 28 at: 1298550634326
    Starting command: 29 at: 1298550634327
    Starting command: 30 at: 1298550634332
    Starting command: 33 at: 1298550634401
    Starting command: 34 at: 1298550634402
    Starting command: 38 at: 1298550634402
    Starting command: 37 at: 1298550634403
    Starting command: 39 at: 1298550634403
    Starting command: 35 at: 1298550634402
    Starting command: 32 at: 1298550634403
    Starting command: 36 at: 1298550634404
    Starting command: 40 at: 1298550634405
    Starting command: 31 at: 1298550634405
    Starting command: 41 at: 1298550634412
    Starting command: 46 at: 1298550634467
    Starting command: 45 at: 1298550634467
    Starting command: 47 at: 1298550634467
    Starting command: 44 at: 1298550634564
    Starting command: 42 at: 1298550634565
    Starting command: 43 at: 1298550634566
    Starting command: 48 at: 1298550634567
    Starting command: 49 at: 1298550634604
    Starting command: 50 at: 1298550635013
    Command finished: 1 at: 1298550635221 (time diff: ) 1038
    Command finished: 3 at: 1298550635225 (time diff: ) 1040
    Starting command: 51 at: 1298550635227
    Starting command: 52 at: 1298550635277
    Command finished: 2 at: 1298550635279 (time diff: ) 1095
    Command finished: 8 at: 1298550635280 (time diff: ) 1043
    Command finished: 14 at: 1298550635298 (time diff: ) 1037
    Command finished: 23 at: 1298550635310 (time diff: ) 1021
    Command finished: 27 at: 1298550635328 (time diff: ) 1020
    Starting command: 53 at: 1298550635337
    Command finished: 29 at: 1298550635364 (time diff: ) 1037
    Command finished: 30 at: 1298550635388 (time diff: ) 1056
    Command finished: 28 at: 1298550635403 (time diff: ) 1077
    Command finished: 31 at: 1298550635434 (time diff: ) 1029
    Command finished: 36 at: 1298550635471 (time diff: ) 1067
    Command finished: 47 at: 1298550635493 (time diff: ) 1026
    Command finished: 45 at: 1298550635513 (time diff: ) 1046
    Command finished: 46 at: 1298550635536 (time diff: ) 1069
    Command finished: 41 at: 1298550635565 (time diff: ) 1153
    Command finished: 48 at: 1298550635586 (time diff: ) 1019
    Command finished: 43 at: 1298550635595 (time diff: ) 1029
    Command finished: 42 at: 1298550635626 (time diff: ) 1061
    Command finished: 44 at: 1298550635640 (time diff: ) 1076
    Command finished: 40 at: 1298550635647 (time diff: ) 1242
    Starting command: 54 at: 1298550635664
    Command finished: 32 at: 1298550635688 (time diff: ) 1285
    Command finished: 35 at: 1298550635694 (time diff: ) 1292
    Command finished: 39 at: 1298550635704 (time diff: ) 1301
    Command finished: 37 at: 1298550635725 (time diff: ) 1322
    Command finished: 34 at: 1298550635747 (time diff: ) 1345
    Command finished: 33 at: 1298550635767 (time diff: ) 1366
    Command finished: 26 at: 1298550635788 (time diff: ) 1480
    Command finished: 25 at: 1298550635808 (time diff: ) 1501
    Command finished: 24 at: 1298550635825 (time diff: ) 1519
    Command finished: 9 at: 1298550635842 (time diff: ) 1554
    Command finished: 21 at: 1298550635868 (time diff: ) 1581
    Command finished: 22 at: 1298550635876 (time diff: ) 1589
    Command finished: 20 at: 1298550635900 (time diff: ) 1613
    Command finished: 19 at: 1298550635917 (time diff: ) 1636
    Command finished: 12 at: 1298550635949 (time diff: ) 1688
    Command finished: 11 at: 1298550635969 (time diff: ) 1706
    Command finished: 13 at: 1298550635992 (time diff: ) 1730
    Command finished: 10 at: 1298550636012 (time diff: ) 1751
    Command finished: 50 at: 1298550636038 (time diff: ) 1025
    Command finished: 18 at: 1298550636056 (time diff: ) 1794
    Command finished: 17 at: 1298550636073 (time diff: ) 1812
    Command finished: 15 at: 1298550636093 (time diff: ) 1833
    Command finished: 7 at: 1298550636107 (time diff: ) 1875
    Command finished: 6 at: 1298550636120 (time diff: ) 1934
    Command finished: 5 at: 1298550636145 (time diff: ) 1959
    Command finished: 4 at: 1298550636163 (time diff: ) 1978
    Command finished: 16 at: 1298550636187 (time diff: ) 1926
    Command finished: 38 at: 1298550636213 (time diff: ) 1811
    Command finished: 49 at: 1298550636225 (time diff: ) 1621
    Command finished: 51 at: 1298550636283 (time diff: ) 1056
    Command finished: 52 at: 1298550636304 (time diff: ) 1027
    Command finished: 53 at: 1298550636366 (time diff: ) 1029
    Command finished: 54 at: 1298550636692 (time diff: ) 1028
    Starting command: 55 at: 1298550637630
    Starting command: 56 at: 1298550637631
    Command finished: 55 at: 1298550638653 (time diff: ) 1023
    Command finished: 56 at: 1298550638682 (time diff: ) 1051
    Starting command: 57 at: 1298550639774
    Starting command: 58 at: 1298550639902
    Starting command: 59 at: 1298550640068
    Starting command: 60 at: 1298550640187
    Command finished: 57 at: 1298550640796 (time diff: ) 1022
    Command finished: 58 at: 1298550640923 (time diff: ) 1021
    Command finished: 59 at: 1298550641088 (time diff: ) 1020
    Command finished: 60 at: 1298550641209 (time diff: ) 1022
    Starting command: 61 at: 1298550641369
    Command finished: 61 at: 1298550642393 (time diff: ) 1024






    and when the input floods:
    Starting command: 3041 at: 1298550654929
    Starting command: 3040 at: 1298550654929
    Starting command: 3039 at: 1298550654929
    Starting command: 3037 at: 1298550654929
    Starting command: 3035 at: 1298550654928
    Starting command: 3033 at: 1298550654928
    Starting command: 3031 at: 1298550654928
    Starting command: 3029 at: 1298550654928
    Starting command: 3027 at: 1298550654928
    Starting command: 3025 at: 1298550654928
    Starting command: 3023 at: 1298550654927
    Starting command: 3019 at: 1298550654927
    Starting command: 3017 at: 1298550654927
    Starting command: 3015 at: 1298550654927
    [I KILLED THE INPUT PROCESS HERE]
    Command finished: 2554 at: 1298550655408 (time diff: ) 1520
    Command finished: 2960 at: 1298550655900 (time diff: ) 1608
    Command finished: 2701 at: 1298550656127 (time diff: ) 1939
    Command finished: 3012 at: 1298550656376 (time diff: ) 1436
    Command finished: 3032 at: 1298550656612 (time diff: ) 1681
    Command finished: 3042 at: 1298550656845 (time diff: ) 1915
    Command finished: 3049 at: 1298550657075 (time diff: ) 2144
    Command finished: 3036 at: 1298550657263 (time diff: ) 2333
    Command finished: 3038 at: 1298550657531 (time diff: ) 2601
    Command finished: 3047 at: 1298550657748 (time diff: ) 2818
    Command finished: 3052 at: 1298550657941 (time diff: ) 3010
    Command finished: 3053 at: 1298550658227 (time diff: ) 3293
    Command finished: 3021 at: 1298550658447 (time diff: ) 3517
    Command finished: 3051 at: 1298550658643 (time diff: ) 3711
    Command finished: 3023 at: 1298550658875 (time diff: ) 3948
    Command finished: 3050 at: 1298550659271 (time diff: ) 4340
    Command finished: 3054 at: 1298550659291 (time diff: ) 4359
    Command finished: 3034 at: 1298550659550 (time diff: ) 4618
    Command finished: 3043 at: 1298550659772 (time diff: ) 4843
    Command finished: 3020 at: 1298550659992 (time diff: ) 5063
    Command finished: 3048 at: 1298550660198 (time diff: ) 5268
    Command finished: 3039 at: 1298550660473 (time diff: ) 5544
    Command finished: 3044 at: 1298550660686 (time diff: ) 5756
    Command finished: 3031 at: 1298550660939 (time diff: ) 6011
    Command finished: 3041 at: 1298550661135 (time diff: ) 6206
    Command finished: 3035 at: 1298550661345 (time diff: ) 6417
    Command finished: 3040 at: 1298550661579 (time diff: ) 6650
    Command finished: 3027 at: 1298550661827 (time diff: ) 6899
    Command finished: 3037 at: 1298550662055 (time diff: ) 7126
    Command finished: 3025 at: 1298550662290 (time diff: ) 7362
    Command finished: 3033 at: 1298550662502 (time diff: ) 7574
    Command finished: 3019 at: 1298550662722 (time diff: ) 7795
    Command finished: 3017 at: 1298550663037 (time diff: ) 8110
    Command finished: 3015 at: 1298550663474 (time diff: ) 8547
    Command finished: 3013 at: 1298550663475 (time diff: ) 8536
    Command finished: 3014 at: 1298550663675 (time diff: ) 8736
    Command finished: 3016 at: 1298550663922 (time diff: ) 8983
    Command finished: 3018 at: 1298550664154 (time diff: ) 9216
    Command finished: 3022 at: 1298550664554 (time diff: ) 9616
    Command finished: 3045 at: 1298550664574 (time diff: ) 9636
    Command finished: 3024 at: 1298550664823 (time diff: ) 9886
    Command finished: 3026 at: 1298550665058 (time diff: ) 10121
    Command finished: 3028 at: 1298550665328 (time diff: ) 10391
    Command finished: 3030 at: 1298550665517 (time diff: ) 10581
    Command finished: 3011 at: 1298550665745 (time diff: ) 10819
    [...]


    Let me know if you have any ideas ;)
    Last edited by Looserette; 02-24-2011 at 12:49 PM.

  6. #6
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    Have you tried launching thousands of processes using a script instead of Java? I'm sure you'll see similar results. Your OS does not have infinite resources at it's disposal. The more processes you create, the more they must share limited resources, and the way they share is by taking turns....which means things take longer.

    I guess this is exactly what I would expect.

  7. #7
    Looserette is offline Member
    Join Date
    Feb 2011
    Posts
    12
    Rep Power
    0

    Default

    Quote Originally Posted by toadaly View Post
    Have you tried launching thousands of processes using a script instead of Java? I'm sure you'll see similar results. Your OS does not have infinite resources at it's disposal. The more processes you create, the more they must share limited resources, and the way they share is by taking turns....which means things take longer.

    I guess this is exactly what I would expect.
    that's true - but my main problem is that java is launching only one or 2 processes at a time for my perl script

    for the sleep command, it's launching up to 4 sleep commands at a time - not more.


    that's really my current issue:
    java is not even trying to run it in parallel.

    Regards,

  8. #8
    toadaly is offline Senior Member
    Join Date
    Jan 2009
    Posts
    671
    Rep Power
    6

    Default

    Java is more than capable of launching thousands of parallel processes as long as your OS can handle it. I've personally written java code that manages hundreds of parallel processes with no issues whatsoever.

    Whatever is going on is almost certainly a bug in your code. Have you tried very simple things to convince yourself that java works?

    Write a simple main program that does nothign except the following:

    Java Code:
    long startTime = System.currentTimeMillis();
    
    for(int i=0; i<100; i++)
      Runtime.getRuntime().exec("sleepy.pl");
    
    long stopTime = System.currentTimeMillis();
    
    System.out.println((stopTime-startTime));
    If the printout happens reasonably fast, then it successfully launched 100 processes without waiting. That shuld convince you that whatever is goign on is in your code, and you can focus your attention on trying to debug it.

Similar Threads

  1. Replies: 1
    Last Post: 01-07-2011, 07:20 AM
  2. Perl in Java!
    By hakan123 in forum New To Java
    Replies: 5
    Last Post: 12-11-2009, 07:05 PM
  3. [B] How to call Perl script from Java [/B]
    By JavaEmpires in forum Advanced Java
    Replies: 7
    Last Post: 12-19-2008, 03:10 PM
  4. run perl script from java
    By maheshmhs in forum New To Java
    Replies: 0
    Last Post: 12-19-2008, 12:40 PM
  5. Use Perl function with java
    By lenny in forum Advanced Java
    Replies: 1
    Last Post: 08-07-2007, 06:25 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •