java.lang.OutOfMemoryError: unable to create new native thread

[jai135]

Hi,

Actually I am facing a problem in a web application which relates to java.lang.OutOfMemoryError: unable to create new native thread

Deployment details for the application are -
1. Sun Java System Application Server 7
2. Java 1.4.2_05
3. Solaris 9 (OS)
4. RAM 8 GB

There are 3-4 applications deployed on Sun Server 7. Some times we receive "Out Of Memory" Error while displaying jsp's in one of the applications deployed on Sun-One Server 7. When Sun Server is restarted, the problem is resolved temporarily.
Since the issue occurs on production environment, its not feasible to restart the Sun Server regularly.

We have also set the java parameters as -Xms 3584M & -Xmx 3584M i.e 3.5 GB around.If this parameter is modified ,the site goes down.

Please provide your valuable suggestions.

Jai

[DevzAbhi]

If i remember correctly solaris 9 is a 32 bit os .. that means the highest memory available for a process is 2 gb .. try config your mem parameters to -Xms 512M & -Xmx 1024M

which will result in early garbage collection . and leave 1 gb of space for application threads

[DevzAbhi]

PS --> please upgrade your jvm to 1.4.2_11 that is more stable and u will have lesser issues .

[Eranga]

I think he's mentioning about the JDK(it's the best word to use :) ) version. So it's always better to move with the latest version.

[DevzAbhi]

I think he's mentioning about the JDK(it's the best word to use :) ) version. So it's always better to move with the latest version.

Thanks for correction......

[Eranga]

No problem pal. Mistakes can be take at anytime. ;) That's why we are discussing here about different topics all the time.

[jai135]

The production site is running on solaris machine so its not possible to use profiler there.

The logs for sun server depicts some error as follows -

[22/Aug/2008:04:16:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:16:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:17:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:17:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:18:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=32 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=32 FreeResources=1 QueueSize=0
[22/Aug/2008:04:18:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:19:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:19:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:20:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:20:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:21:10] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:21:26] WARNING (22501): for host 172.16.0.108 trying to GET /robots.txt, send-file reports: HTTP4142: can't find /var/opt/SUNWappserver7/domains/domain1/server1/applications/j2ee-modules/AAI_CMSPROD_1/robots.txt (File not found)
[22/Aug/2008:04:21:40] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:22:15] WARNING (22501): CORE3283: stderr: java.lang.OutOfMemoryError
[22/Aug/2008:04:22:20] SEVERE (22501): MONITORING : Pool [aaidb_1] PoolSize=31 FreeResources=0 QueueSize=0, Pool [aaidb_0] PoolSize=31 FreeResources=0 QueueSize=0
[22/Aug/2008:04:22:28] SEVERE (22501): CORE4007: Internal error: Unexpected Java exception thrown (java.lang.OutOfMemoryError, no description), stack: java.lang.OutOfMemoryError

The above logs depict that as soon as PoolSize becomes 31 from 32 , FreeResources also turn 0 from 1 and this throws OutOfMemoryError. Now the server needs to be restarted which makes the PoolSize again as 8 and the site starts working again.

Does this have to do anything with the entries in server.xml which is as follows in my case -

<jdbc-connection-pool steady-pool-size="8" max-pool-size="32" max-wait-time-in-millis="60000" pool-resize-quantity="2" idle-timeout-in-seconds="300" is-isolation-level-guaranteed="false" is-connection-validation-required="false" connection-validation-method="auto-commit" fail-all-connections="false" datasource-classname="oracle.jdbc.pool.OracleDataSource" res-type="javax.sql.XADataSource">

[DevzAbhi]

See poolsize depicts the number of open connection to the data source . you can try reducing the max pool size to 20 if your performance is not hampered ...did you try the the mem size i wrote ..
wat i can make out from log is the connection to data base require some memory which is not available due to high mem size currently configured .

[DevzAbhi]

and the limitation of os to allocate only 2 gb

[masijade]

Solaris 9 is also available in 64 bit. In fact, if I remember right, that is the default as long as the machine it is being installed on can do 64 bit. Also, two gig would not be the limit only to one process, but the system as a whole (unless it is running on a Sun Fire server, where each processor has it's own memory management module, but then it would have to be 64 bit anyway, since all sun Fire servers are purely 64 bit).

As far as I know, you can also get Solaris 8 in 64 bit (but I could be wrong about that one).

[jai135]

Hi,

I reduced the heap size to Xms=512 Xmx=1024.But still I recieved outOfMemory Error.

[01/Sep/2008:13:12:26] SEVERE (17913): MONITORING : Pool [aaidb_1] PoolSize=8 FreeResources=1 QueueSize=0, Pool [aaidb_0] PoolSize=8 FreeResources=1 QueueSize=0
[01/Sep/2008:13:12:28] INFO (17913): CORE3282: stdout: Session Bhadiya..
[01/Sep/2008:13:12:28] INFO (17913): CORE3282: stdout: Action : tenders_cancel
[01/Sep/2008:13:12:28] INFO (17913): CORE3282: stdout: Inside Get Modeltenders_cancel
[01/Sep/2008:13:12:33] SEVERE (17913): StandardWrapperValve[jsp]: Servlet.service() for servlet jsp threw exception
javax.servlet.ServletException
at com.iplanet.ias.web.jsp.JspServlet.service(JspServ let.java:296)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)
at org.apache.catalina.core.StandardWrapperValve.invo keServletService(StandardWrapperValve.java:720)
at org.apache.catalina.core.StandardWrapperValve.acce ss$000(StandardWrapperValve.java:118)
at org.apache.catalina.core.StandardWrapperValve$1.ru n(StandardWrapperValve.java:278)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:274)
at org.apache.catalina.core.StandardPipeline.invoke(S tandardPipeline.java:505)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:212)
at org.apache.catalina.core.StandardPipeline.invoke(S tandardPipeline.java:505)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:203)
at org.apache.catalina.core.StandardPipeline.invoke(S tandardPipeline.java:505)
at com.iplanet.ias.web.connector.nsapi.NSAPIProcessor .process(NSAPIProcessor.java:158)
at com.iplanet.ias.web.WebContainer.service(WebContai ner.java:856)
----- Root Cause -----
java.lang.OutOfMemoryError

[01/Sep/2008:13:12:35] SEVERE (17913): StandardWrapperValve[jsp]: Servlet.service() for servlet jsp threw exception
javax.servlet.ServletException
at com.iplanet.ias.web.jsp.JspServlet.service(JspServ let.java:296)
at javax.servlet.http.HttpServlet.service(HttpServlet .java:853)
at org.apache.catalina.core.StandardWrapperValve.invo keServletService(StandardWrapperValve.java:720)
at org.apache.catalina.core.StandardWrapperValve.acce ss$000(StandardWrapperValve.java:118)
at org.apache.catalina.core.StandardWrapperValve$1.ru n(StandardWrapperValve.java:278)
at java.security.AccessController.doPrivileged(Native Method)
at org.apache.catalina.core.StandardWrapperValve.invo ke(StandardWrapperValve.java:274)
at org.apache.catalina.core.StandardPipeline.invoke(S tandardPipeline.java:505)
at org.apache.catalina.core.StandardContextValve.invo ke(StandardContextValve.java:212)
at org.apache.catalina.core.StandardPipeline.invoke(S tandardPipeline.java:505)
at org.apache.catalina.core.StandardHostValve.invoke( StandardHostValve.java:203)
at org.apache.catalina.core.StandardPipeline.invoke(S tandardPipeline.java:505)
at com.iplanet.ias.web.connector.nsapi.NSAPIProcessor .process(NSAPIProcessor.java:158)
at com.iplanet.ias.web.WebContainer.service(WebContai ner.java:856)
----- Root Cause -----
java.lang.OutOfMemoryError


Is there any other configuration that I can try to eliminate this error

Regards
Jai

[DevzAbhi]

please upgrade to jdk 1.4.2_11 i hope it helps.

because i have faced the issue .. on windows .. was able to resolve it after updating the JDK