Results 1 to 9 of 9
  1. #1
    viking is offline Member
    Join Date
    Dec 2010
    Posts
    7
    Rep Power
    0

    Post ScheduledThreadPool Executor Problem

    Hi All,

    I have used ScheduledThreadPool Executor class to perform some task on a database and this task has to be done every minute and it should go on for ever.

    The main thread created will create child threads and the actual work is done by these worker threads.

    The problem that i'm facing is the whole process runs properly for around 4 days and after that the programs stops executing without any errors.

    The logs show that the control is stuck with the worker thread but the thread is not throwing any error.

    Can someone tell me if this is a problem with threads?

  2. #2
    travishein's Avatar
    travishein is offline Senior Member
    Join Date
    Sep 2009
    Location
    Canada
    Posts
    684
    Rep Power
    6

    Default

    Ah neat. I didn't know ScheduledThreadPool executors existed before now, and I am going to read up on them.

    I used to get crazy null pointer errors showing up in other log files in tomcat , for example, my application had its own log file, but there is also catalina.out and the localhost-yyyy-mm-dd.log file. I never thought to look in those last two log files, I assumed since i was using my own logger everything would end up in my log file.

    Another idea, are the treads really exited, or just in a deadlock, the next time it gets into that state, would you be able to get a thread dump from the VM (issue a "kill -3" to the java vm process)? for example, if I start tomcat on a unix box, and use ps to find the process ID, I can issue "kill -3 $pid" and the catalina.out contains an eye watering amount of stack trace looking mess that details what each thread is doing. It might be possible to see if the threads are hung waiting on something (there will be line numbers in the dump), or if the threads are just gone (exited).

    I find it helpful to put log.debug messages at the beginning and end of all my thread run() methods to tell me "thread x starting". and thread x exiting.".

    another idea, for web / server applications, I have had great success using the Quartz job scheduler framework. It is a single jar file to add, and can be configured using Spingframework, or stand alone with a properties file.

    I used to make my own threaded service kind of class to boot up in web application when it started up (and shut down on stop) using servlet context listeners. However later on when I started having my applications deployed in a cluster, we observed the problem that each node that had an instance of the application would each run the scheduled thread every (few seconds or when ever it was supposed to run). This was not critical, but was not ideal because it was wasting resources querying the same data for the same update operation, and it might cause confusing system states if it ever chewed up the same data at the same time.

    So when I switched it to use Quartz, they support using a database with tables for storing the job execution state, where each node in the cluster works with the quartz database in an orderly manner, so that the job is ran on exactly one node at a time as often as it is needed. it also has built in fail over , shutting down one node a different node would run the scheduled task at the exact moment it was supposed to be ran, didn't miss a beat.

    As you can tell I am very excited. Not sure if its possible to switch to Quartz in your case, but if you can, it might be helpful.

  3. #3
    viking is offline Member
    Join Date
    Dec 2010
    Posts
    7
    Rep Power
    0

    Post

    travishein,

    Thanks for the reply. So if i use kill -3 on a process where can i find the logs? What i mean where can i find the logs of the java VM process that i have to kill?

  4. #4
    travishein's Avatar
    travishein is offline Senior Member
    Join Date
    Sep 2009
    Location
    Canada
    Posts
    684
    Rep Power
    6

    Default

    if this is in tomcat, usually inside the logs folder there is a "catalina.out" on windows there might be some other log file that the windows service wrapper redirects them out to .

    i'm not familiar with other application servers, but there is always some kind of a log file somewhere. the thread dump is printed to the standard output (or standard error) of the java vm, so where ever that is being redirected to.

    some application servers such as tomcat, allow you to start the process in the shell script wrapper inside a terminal window and keep it in the foreground (not redirecting the standard to some log file). that might help to see the thread dump too.

  5. #5
    viking is offline Member
    Join Date
    Dec 2010
    Posts
    7
    Rep Power
    0

    Post

    travishein,

    This is not a web application running on a web server. Its just a plain java application which is using threads, so is there any way i can find out the logs ?

  6. #6
    travishein's Avatar
    travishein is offline Senior Member
    Join Date
    Sep 2009
    Location
    Canada
    Posts
    684
    Rep Power
    6

    Default

    well, find out how it is started. If it uses a shell script, maybe they do something like
    Java Code:
    java -jar myapp.jar > logs/my.log
    where it is redirecting output to some log file.

    Maybe the application uses a logging framework, where it is outputting messages to a configured log file.

    if it uses something like log4j or logback or java util logger, there might be some file log4j.xml log4j.properties logback.xml. or something in the shell script that sets -Djava.util.logging.config= in the java vm arguments.

    It is possible the logging configuration file is buried inside one of the application jar file, or a classes/ folder. It's also possible the log configuration is set to some level to only emit error messages.

    It's also possible the application was coded with empty try...catch blocks so the exception is not even being logged anywhere when it happens.

  7. #7
    lovelesh is offline Senior Member
    Join Date
    Nov 2010
    Location
    Delhi
    Posts
    135
    Blog Entries
    1
    Rep Power
    0

    Default

    Hi Viking

    The problem is not with threads. Its with your heap memory. Your application must be behaving as if it is dying, right?

    If yes, then this is happening because you might have forgotten to close statement after doing database operation.
    As you didn't close the statement, the heap memory remains allocated to Connection (actually statement object). Once all the memory is consumed, you application starts behaving like this.

  8. #8
    viking is offline Member
    Join Date
    Dec 2010
    Posts
    7
    Rep Power
    0

    Default

    lovelesh,

    Thanks for the tip. I encountered an error now. It is as below.


    2010-12-23 09:47:38.575 java[31924:1707] *** __NSAutoreleaseNoPool(): Object 0x10015b230 of class NSConcreteMapTableValueEnumerator autoreleased with no pool in place - just leaking

    Can you please let me know if it is related to heap memory problem?

  9. #9
    viking is offline Member
    Join Date
    Dec 2010
    Posts
    7
    Rep Power
    0

    Post

    lovelesh,

    You were spot on with. The connections used were not freed. Now its working fine. Thanks the issue is resolved.

Similar Threads

  1. Interface and executor
    By hannes in forum AWT / Swing
    Replies: 2
    Last Post: 01-03-2010, 08:37 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •