HashMap question

[new_2_java]

Hi all,

I have an HashMap with key/value pair. The program processes each element of the hashMap, and should mark the HashMap element as "processed". i.e. I have the follwoing key/value pair.

1/"090003453bc454"
2/"090003453bc455"
3/"090003453bc456"
and so on...

So, when the program process each element, it should update the value to "Processed". So, each process element should look like the following.

1/"Processed"
2/"Processed"
3/"Processed"
and so on...

So, my question is, how can I re-set the value from orginal value of "0900..." ot "Processed"

Please advise...

Thanks,

[new_2_java]

Can I use put(...) method of HashMap to re-set a value for a specified key? i.e.

HashMap map = new HashMap();
map.put(new Integer(1), new String("Value1");
map.put(new Integer(2), new String ("Value2");

// now I want to re-set the value in key 1 to something else. i.e.

map.put(new Integer(1), new String("Processed");

Will this over-write "Value1" to "Processed" at key 1

OR

Will it simply add another key/value pair of 1/"Processed"

Please clarify...

Thanks,

[ojn]

Can I use put(...) method of HashMap to re-set a value for a specified key? i.e.

Try and see for yourself? If you're unsure how to use put(), check the API.

[new_2_java]

Yes, you are right. It's simplest way. :-)

I would like to ask another side question.


Senario:


I have a threaded program which operates on this HashMap values. So, taking the above as an example where I have key/values in my HashMap.

The program will spawn lets say 10 threads. Each thread will look at the next element of HashMap and verifies whether the value for that key is marked as "Processed", if it's not marked proceesed, first it marks that element as "Processed" so the other threads will skip this particular element. Then it will process the element.

i.e. Here's how the skeleton of the program will look like

public myClass extends Thread {
    public static void main(String[] arg) {
        myClass m = new myClass ();
        HashMap map = m.loadHashMap ();  // this will only load the HashMap wil key/value as mentioned above
        for (int i =0; i < 10; i++) {
            m.start();
        } 
    }
    
    public void run () {
        for (int j=0; j < map.size(); j++) {
            if (!map.get(new Integer(j)).toString().equalsIgnoreCase("Processed")) {
                map.put(new Integer(j), new String("Processed");
                processElement(map.get(new Integer(j)).toString()); // this will the actual process
            }
        }
    }
}

The reason I am trying to this way is because I am operating on a very large amount of data. I am using thread to reduce the amount of time that the program takes.

So, can someone comment as whether this works? Whether all the threads process each element of the HashMap according to what I have designed? Basically, I am trying to simulate pasemistic locking mechanism.

Please advise...

Thanks,

[ojn]

You can also shorten the put() call if you want to.

map.put(1, "Value1");

works fine.

[new_2_java]

I think, HashMap.put(..) expects an Object as input parameter and not int type.

[ojn]

Java wraps you primitive type (int) in an Integer (object) automatically. This is called autoboxing. This has been possible since Java >= 5.

[new_2_java]

I use 1.4.2, maybe that's why I get compilation error.

But anyways, my concern was about my approach. Whether it will work or not.

If someone can comment on this, as apposed to "try and see it for yourself", as the program is quite complex and can't test it untill I make an effort/design and code to just to a test.

Thanks,

[Norm]

Sometimes the quickest and easiest way is to write a short one page program to test a coding technique. Fast and easy to change and test. When you've figured out how a class works, you can then use that technique in the main project with confidence.

Can you explain what pasemistic locking is?

What happens in the code you posted when you start() the same thread more than one time?

As you progress into the HashMap you are going to spend a lot of time skipping over processed entries. Rethink the access to allow sequential access and then assign ranges to each of the threads.

[ojn]

I use 1.4.2, maybe that's why I get compilation error.

Yeah, then autoboxing won't work.

But anyways, my concern was about my approach. Whether it will work or not.

If someone can comment on this, as apposed to "try and see it for yourself", as the program is quite complex and can't test it untill I make an effort/design and code to just to a test.

Thanks,

Just write a small test program that does just that. Adds a value to the HashMap, prints it, then adds it again - same key, different value - and prints the updated one. Then you'll see a before and after "picture".

[fishtoprecords]

I have a threaded program which operates on this HashMap values. So, taking the above as an example where I have key/values in my HashMap.

The program will spawn lets say 10 threads.

Whenever you have more than one thread looking at a structure, you have to be thread safe.

When you read the javadocs for HashMap it says in bold letters "this is not synchromized"

[Norm]

I am using thread to reduce the amount of time that the program takes.

Are there parts of the work being done that can be concurrent with other parts? Do you have a mulitprocessor machine? If not, then threads will increase the time for the job. There will be no advantage in having multiple threads if they aren't doing something at the same time.
Doing them all in one thread will be the fastest if there is no possibility of overlapped processing.
Swapping from one thread to another thread will add to the time.

[new_2_java]

Norm,

I am writing a program that will be used to migrate content/metadata from the source repository to a target repository. The over-head will be the movement of the contetn. In other word the program will be I/O intensive, and not CPU. So, I believe--and correct me if I am wrong--multi-thread will make a difference.

the content is somewhere arround 40 to 50 GB.

Here's what I have tried, and, since I am not very versed with threads, I would like experts to suggest.

public class MigrationUtility {
    private HashMap map = new HashMap ();
    
    public static void main(String[] args){
        MigrationUtility m = new MigrationUtility ();
        m.loadMap();
        
        for (int j=0; j < 10; j++) {
            new Thread(new MigrationUtilityThread(m.map)).start();
        }        
    }
    
    private void loadMap () {
	// This is just a dummy values for test purpose only.
	// Values will eventually be populated properly.
        for (int i = 0;i < 50; i++) {
            map.put(new Integer(i), new String("Value1" + i));
        }
    }
}

class MigrationUtilityThread extends Thread{
    private HashMap m = null;
    MigrationUtilityThread(HashMap map){
        m = map;
    }
    
    public void run(){
        for (int j=0; j < m.size(); j++) {
            if (!m.get(new Integer(j)).toString().equalsIgnoreCase("Processed")) {
                m.put(new Integer(j), new String("Processed"));
                processItem(this.getName());                
            }
        }
    }
    
    public synchronized void processItem(String name) {
	// The actual process will happen here...
        System.out.println("Item processed with thread: "+ name );
    }
}

The output of this program is as follow:

Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-8
Item processed with thread: Thread-6
Item processed with thread: Thread-10
Item processed with thread: Thread-18
Item processed with thread: Thread-2
Item processed with thread: Thread-4
Item processed with thread: Thread-12
Item processed with thread: Thread-16
Item processed with thread: Thread-0
Item processed with thread: Thread-14

It looks like different threads can operate on the map.

Pasemistic locking is that you lock an item first, and then process it, where as optemistic locking is you process an item first, and after the process is successful, then you lock it.

Thanks again, and waiting for reply...

[Norm]

To see if multiple threads will help, do a timing test for each case and see which is faster. If the I/Os are independent and can be concurrent, then multiple threads should help.

Instead of having all the threads look at all of the entries in the Hashmap and overwriting them for a flag, get all the keys in the Hashmap in a list and split them up, giving a range of keys in the list to each thread. Then you would not need to update the Hashmap, you would only be reading it.

[new_2_java]

Norm, thanks for the suggestions. Here's what I have tried so far, just as a template program, however, I get un-expected results. Can you take a look and comment as why its behaving this way.

public class MigrationUtility {
    private HashMap map = new HashMap ();
    private List list1 = new ArrayList ();
    private List list2 = new ArrayList ();
    private List list3 = new ArrayList ();
    private List list4 = new ArrayList ();
    
    public static void main(String[] args){
        MigrationUtility m = new MigrationUtility ();
        m.loadMap();
        
        
            new Thread(new MigrationUtilityThread(m.map, m.list1)).start();
            new Thread(new MigrationUtilityThread(m.map, m.list2)).start();
            new Thread(new MigrationUtilityThread(m.map, m.list3)).start();
            new Thread(new MigrationUtilityThread(m.map, m.list4)).start();
        
    }
    
    private void loadMap () {

        for (int i = 0;i < 40; i++) {
            map.put(new Integer(i), new String("Value1" + i));
            if (i < 10) {                
                if (!list1.add(new Integer(i))) {
                    return;
                }
            } else if (i < 20) {
                if (!list2.add(new Integer(i))) {
                    return;
                }
            } else if (i < 30) {
                if (!list3.add(new Integer(i))) {
                    return;
                }                
            } else {
                if (!list4.add(new Integer(i))) {
                    return;
                }
            }
        } 
    }
}

class MigrationUtilityThread extends Thread{
    private HashMap m = null;
    private List list = null;
    MigrationUtilityThread(HashMap map, List list){
        m = map;
        this.list = list;
    }
    
    public void run() {
        
        for (int j=0; j < list.size(); j++) {
            synchronized (m) {
                if (!m.get(list.get(j)).toString().equalsIgnoreCase("Processed")) {
                    m.put(new Integer(j), new String("Processed"));
                    processItem(this.getName(), list.get(j), m);                
                }
            }
        }
    }
    
    public synchronized void processItem(String name, Object j, HashMap m) {
        System.out.println("Item processed with thread: "+ name + " indedx of: " + j + " Map item: " + m.get(j));
    }
}

The output is the following:

Item processed with thread: Thread-0 indedx of: 0 Map item: Processed
Item processed with thread: Thread-0 indedx of: 1 Map item: Processed
Item processed with thread: Thread-0 indedx of: 2 Map item: Processed
Item processed with thread: Thread-0 indedx of: 3 Map item: Processed
Item processed with thread: Thread-0 indedx of: 4 Map item: Processed
Item processed with thread: Thread-0 indedx of: 5 Map item: Processed
Item processed with thread: Thread-0 indedx of: 6 Map item: Processed
Item processed with thread: Thread-0 indedx of: 7 Map item: Processed
Item processed with thread: Thread-0 indedx of: 8 Map item: Processed
Item processed with thread: Thread-0 indedx of: 9 Map item: Processed
Item processed with thread: Thread-4 indedx of: 20 Map item: Value120
Item processed with thread: Thread-4 indedx of: 21 Map item: Value121
Item processed with thread: Thread-4 indedx of: 22 Map item: Value122
Item processed with thread: Thread-4 indedx of: 23 Map item: Value123
Item processed with thread: Thread-4 indedx of: 24 Map item: Value124
Item processed with thread: Thread-4 indedx of: 25 Map item: Value125
Item processed with thread: Thread-4 indedx of: 26 Map item: Value126
Item processed with thread: Thread-4 indedx of: 27 Map item: Value127
Item processed with thread: Thread-4 indedx of: 28 Map item: Value128
Item processed with thread: Thread-4 indedx of: 29 Map item: Value129
Item processed with thread: Thread-6 indedx of: 30 Map item: Value130
Item processed with thread: Thread-6 indedx of: 31 Map item: Value131
Item processed with thread: Thread-6 indedx of: 32 Map item: Value132
Item processed with thread: Thread-6 indedx of: 33 Map item: Value133
Item processed with thread: Thread-6 indedx of: 34 Map item: Value134
Item processed with thread: Thread-6 indedx of: 35 Map item: Value135
Item processed with thread: Thread-6 indedx of: 36 Map item: Value136
Item processed with thread: Thread-6 indedx of: 37 Map item: Value137
Item processed with thread: Thread-6 indedx of: 38 Map item: Value138
Item processed with thread: Thread-6 indedx of: 39 Map item: Value139
Item processed with thread: Thread-2 indedx of: 10 Map item: Value110
Item processed with thread: Thread-2 indedx of: 11 Map item: Value111
Item processed with thread: Thread-2 indedx of: 12 Map item: Value112
Item processed with thread: Thread-2 indedx of: 13 Map item: Value113
Item processed with thread: Thread-2 indedx of: 14 Map item: Value114
Item processed with thread: Thread-2 indedx of: 15 Map item: Value115
Item processed with thread: Thread-2 indedx of: 16 Map item: Value116
Item processed with thread: Thread-2 indedx of: 17 Map item: Value117
Item processed with thread: Thread-2 indedx of: 18 Map item: Value118
Item processed with thread: Thread-2 indedx of: 19 Map item: Value119

As you can see, only the first thread updates the map values to "Processed" and the rest of the map values stays the same. Can someone advise as where I am going wrong?

Thanks in advance...

[Norm]

I think your design should be changed. Instead of having the threads access/maintain the HashMap, you should have a method to control that.
Here's a version that has the threads read files from a folder as a job.

// Using threads to process contents of a HashMap
// Use dispatcher to give out jobs vs each task getting their own

import java.util.*;
import java.io.*;

public class MigrationUtility2 {
    private HashMap map = new HashMap ();


    // Define controlling variables:
    final int BuffSize = 4096;
    final int ModulusYieldFactor = 0; // yield once every ..
    final int NbrElements = 100;
    final int NbrThreads  = 10; 
    final int ProcessEach = NbrElements / NbrThreads;

    private int threadCnt = 0;
    public  synchronized void changeThreadCnt(int amt) {
       threadCnt += amt;
       if(threadCnt == 0)
         notify();
    }
    public synchronized int getThreadCnt() {
       return threadCnt;
    }
    
    Integer[] keys;
    int nextKey = 0;

    //---------------------------------------------------------
    public static void main(String[] args){
       try{
        MigrationUtility2 m = new MigrationUtility2 ();
       }catch(Exception x) {
         x.printStackTrace();
       }
    } // end main()

    public MigrationUtility2() throws Exception {
        loadMap();                     // put some data in the map to be processed
        
        Set set = map.keySet();
        keys = (Integer[])set.toArray(new Integer[0]);
        Thread t = null;
        long start = System.currentTimeMillis();  // save starting time
        // Start threads to process each range of keys
        for (int j=0; j < NbrThreads; j++) {
          t = new Thread(new MigrationUtilityThread2(this));
          t.start();
        } // end for() starting all work threads    

        // Wait for all threads to end
        synchronized(this) { 
            wait();
        }
        System.out.println("Exiting constructor dur=" + (System.currentTimeMillis()-start)
                           + ", nbrThread=" + NbrThreads + ", nbrElem=" + NbrElements
                           + ", yield=" + ModulusYieldFactor); 
    } // end Constructor

    //-----------------------------------------------------
    // Return next job to do
    public synchronized String getNextJob(){
      if(nextKey < keys.length)
         return (String)map.get(keys[nextKey++]);
      else 
         return null;
    }

    //------------------------------------------------------------------------------------------
    // Create some work to do - read all files on each path
    final String[] Paths = new String[] {"C:\\BatchFiles", "D:\\BatchFiles", "E:\\My_Photos",
                                          "F:\\RegistryBackups", "G:\\Miscellaneous\\SailnetArchives",
                                          "H:\\Backups\\Data (E)\\NoonSite"};

    private void loadMap () {
	// This is just a dummy values for test purpose only.
	// Values will eventually be populated properly.
        int ix = 0;
        for (int i = 0;i < NbrElements; i++) {
            map.put(new Integer(i), Paths[ix++]);
            if(ix >= Paths.length)  ix = 0;
        }
    }
} // end class MigrationUtility

//-------------------------------------------------------------------------
// Runs thread to do some work
class MigrationUtilityThread2 extends Thread{
    static int cnt = 0;                // Count items processed
    private int jobsDone = 0;

    MigrationUtility2 mu;


    MigrationUtilityThread2(MigrationUtility2 mu){
        this.mu = mu;
        mu.changeThreadCnt(+1);        // Log immediately
    }
    
    public void run(){
        System.out.println("Startin run() for " + this.getName());
        while(true) {
            String value = mu.getNextJob(); // get next job to do
            if (value == null) {
               break;                  // exit if none left
            }
            processItem(value, this.getName());  // Go process this job
            jobsDone++;                // Count nbr jobs done      
        } // end while(true)
        mu.changeThreadCnt(-1);
        System.out.println(">>> Exiting " + this.getName() + " nbr jobs=" + jobsDone);
    }

    
    private void processItem(String value, String name) {
	   // The actual process will happen here...
      int bytesRead = 0;
      try {
         File f = new File(value);
         String[] files = f.list();
         if(files == null || files.length == 0) {
            System.out.println(">>>>>>>>>>>>> NO FILEs FOR " + value);
            return;
         }
         for(int i=0; i < files.length; i++) {
            File fi = new File(f, files[i]);
            if(fi.isDirectory()) { // || !fi.exists()) {
//               System.out.println("skipping " + fi + " " + fi.exists());
               continue;                  // skip these
            }
            FileInputStream fis = new FileInputStream(fi);
            byte[] buff = new byte[4096]; //mu.BuffSize];
            int nbrRead = 0;
            int  ix = 0;
            while((nbrRead = fis.read(buff)) > 0) {
               bytesRead += nbrRead;
               if(mu.ModulusYieldFactor > 1 && (ix++ % mu.ModulusYieldFactor == 0)) {
                  Thread.yield();   // SLOWS IT DOWN!!!
               }   
            }
            fis.close();
         } // end for
      }catch(Exception x) {
         x.printStackTrace();
      }
      // Show results
      System.out.println(++cnt + " Item " + value + " processed with thread: "+ name 
                        + ", read " + bytesRead);
    }
} // end class MigrationUtilityThread

[new_2_java]

Norm,

Thanks very much. Real appreciate that. I can take it as my starting point and work arround that. This is a great help.

Thanks once again.

[new_2_java]

Norm,

Just a quick question. Is there a reason, why create all the threads in the constructor? as apposed to main method?

In other words, is it ok to do the following?

    public static void main(String[] args){
       try{
        MigrationUtility3 m = new MigrationUtility3 ();
	// The reason is because I want to do some pre-processing tasks, before
	// spawning the threads...
	m.createThreads();
       }catch(Exception x) {
         x.printStackTrace();
       }
    } // end main()

    public MigrationUtility3() throws Exception {
        loadMap();                     // put some data in the map to be processed
        
        Set set = map.keySet();
        keys = (Integer[])set.toArray(new Integer[0]);
              
    } // end Constructor

    public void createThreads() {
	Thread t = null;
        long start = System.currentTimeMillis();  // save starting time 
	// Start threads to process each range of keys
        for (int j=0; j < NbrThreads; j++) {
          t = new Thread(new MigrationUtilityThread3(this));
          t.start();
        } // end for() starting all work threads    

        // Wait for all threads to end
        synchronized(this) { 
            wait();
        }
        System.out.println("Exiting constructor dur=" + (System.currentTimeMillis()-start)
                           + ", nbrThread=" + NbrThreads + ", nbrElem=" + NbrElements
                           + ", yield=" + ModulusYieldFactor); 
    }

The reason, why I am asking this, is because I want to do some pre-processing tasks, before spawning the threads.

Thanks again.

[new_2_java]

Also, one more question.

If I add some more methods, do I need to "synchonize"d them? I guess not, so long as it doesn't change the content of HashMap, correct me if I am wrong, please.

Thanks,

[Norm]

The code in the constructor references class variables. I prefer main to do the shortest simplest part of the app.
Your m.method(); from main
is the same as
method(); at the end of the constructor.
A reason to move the code outside of the constructor could be if you needed to do something after creating the object and before the method was called.

Use synchronized for methods that are called from threads and that update data.