Results 1 to 2 of 2
  1. #1
    mapred.learn is offline Member
    Join Date
    Mar 2011
    Posts
    3
    Rep Power
    0

    Default how to run parallel processes from a java program ?

    Hi,
    My use case is to split a huge gz file into multiple files and then run another java code in parallel on these multiple files and exit the main program when these all child programs finish.

    My question is
    how can I do it in java ?:
    i) what is best way to split a huge file (20-30 GB) into smaller chunks ?
    ii) what is best way to run lot of parallel processes from a java program, where all processes do the same thing and main process should exit when these processes finish.
    iii) Also, is it possible for main program to get exception if any of the child fails and it can re-submit this child again after correcting error condition .

    Thanks,
    -JJ

  2. #2
    travishein's Avatar
    travishein is offline Senior Member
    Join Date
    Sep 2009
    Location
    Canada
    Posts
    684
    Rep Power
    5

    Default

    have a look at the Java Parallel Processing Framework
    They have an API to facilitate creating workers and message passing infrastructure.

    For reading a large tar.gz file ,that part might be single thread only, as reading a stream generally is sequential. Perhaps what can be done is the large file gets un-tarred into individual files and folders into a temporary or work folder, and as a file is created, an event is fired to say "this is a file that needs working on" and use that File handle as the token or input to the work order to be dispatched to a processing node.

Similar Threads

  1. how to run parallel processes ?
    By mapred.learn in forum New To Java
    Replies: 2
    Last Post: 03-02-2011, 12:45 AM
  2. parallel port programming in java
    By naman.jain14 in forum Networking
    Replies: 0
    Last Post: 02-22-2011, 05:57 PM
  3. Too many java.exe processes due to netbeans
    By abhishek.jain in forum NetBeans
    Replies: 7
    Last Post: 12-29-2009, 01:52 PM
  4. Java Parallel Processing Framework 1.2
    By Java Tip in forum Java Software
    Replies: 0
    Last Post: 05-06-2008, 06:44 PM
  5. Java Parallel Processing Framework 1.0 RC1
    By JavaBean in forum Java Software
    Replies: 0
    Last Post: 11-17-2007, 02:12 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •