java - WebSphere 7 - Can excessive Garbage Collection lead to out of memory? -

issue: getting out of native memory exception , wondering if excessive garbage collection can lead this? advice on gc policy or tuning helpful. i'm not sure if have warrants change yet.

good reference stackoverflow question: which gc policy use

specs:

• server environment: websphere version 7
• gc policy : default ( optthruput )
• java 1.5
• heap: 8 gb
• running in vm
• analysis tool: app dynamics

preliminary analysis:

1. i assumed memory leak garbage collection looks ok reclaims memory
2. native memory exception makes me think memory outside of vm being exhausted, not sure how.

attached screenshot:

1.1 - 4 hour period of heap utilization . each little green garbage can represents major garbage collection point. 1.2 gc time spend in above graph. 1.3 heap utilization looked during out of memory exception.

exception:

ejb threw unexpected (non-declared) ejb exception data: java.lang.outofmemoryerror: native memory exhausted @ garbagecollection.mycode.test @ com.ibm.ws.webcontainer.servlet.servletwrapper.service(servletwrapper.java:1658) @ com.ibm.ws.webcontainer.servlet.servletwrapper.service(servletwrapper.java:1598) @ com.ibm.ws.webcontainer.filter.webappfilterchain.dofilter(webappfilterchain.java:149) @ com.ibm.ws.webcontainer.filter.filterinstancewrapper.dofilter(filterinstancewrapper.java:190) @ com.ibm.ws.webcontainer.filter.webappfilterchain.dofilter(webappfilterchain.java:125) @ com.ibm.ws.webcontainer.filter.filterinstancewrapper.dofilter(filterinstancewrapper.java:190) @ com.ibm.ws.webcontainer.filter.webappfilterchain.dofilter(webappfilterchain.java:125) @ com.ibm.ws.webcontainer.filter.filterinstancewrapper.dofilter(filterinstancewrapper.java:190) @ com.ibm.ws.webcontainer.filter.webappfilterchain.dofilter(webappfilterchain.java:125) @ com.ibm.ws.webcontainer.filter.webappfilterchain._dofilter(webappfilterchain.java:80) @ com.ibm.ws.webcontainer.filter.webappfiltermanager.dofilter(webappfiltermanager.java:908) @ com.ibm.ws.webcontainer.servlet.servletwrapper.handlerequest(servletwrapper.java:935) @ com.ibm.ws.webcontainer.servlet.servletwrapper.handlerequest(servletwrapper.java:503) @ com.ibm.ws.webcontainer.servlet.servletwrapperimpl.handlerequest(servletwrapperimpl.java:181) @ com.ibm.ws.webcontainer.servlet.cacheservletwrapper.handlerequest(cacheservletwrapper.java:91) @ com.ibm.ws.webcontainer.webcontainer.handlerequest(webcontainer.java:875) @ com.ibm.ws.webcontainer.wswebcontainer.handlerequest(wswebcontainer.java:1592) @ com.ibm.ws.webcontainer.channel.wcchannellink.ready(wcchannellink.java:186) @ com.ibm.ws.http.channel.inbound.impl.httpinboundlink.handlediscrimination(httpinboundlink.java:453) @ com.ibm.ws.http.channel.inbound.impl.httpinboundlink.handlenewrequest(httpinboundlink.java:515) @ com.ibm.ws.http.channel.inbound.impl.httpinboundlink.processrequest(httpinboundlink.java:306) @ com.ibm.ws.http.channel.inbound.impl.httpiclreadcallback.complete(httpiclreadcallback.java:83) @ com.ibm.ws.ssl.channel.impl.sslreadservicecontext$sslreadcompletedcallback.complete(sslreadservicecontext.java:1784) @ com.ibm.ws.tcp.channel.impl.aioreadcompletionlistener.futurecompleted(aioreadcompletionlistener.java:165) @ com.ibm.io.async.abstractasyncfuture.invokecallback(abstractasyncfuture.java:217) @ com.ibm.io.async.asyncchannelfuture.firecompletionactions(asyncchannelfuture.java:161) @ com.ibm.io.async.asyncfuture.completed(asyncfuture.java:138) @ com.ibm.io.async.resulthandler.complete(resulthandler.java:204) @ com.ibm.io.async.resulthandler.runeventprocessingloop(resulthandler.java:775) @ com.ibm.io.async.resulthandler$2.run(resulthandler.java:905) @ com.ibm.ws.util.threadpool$worker.run(threadpool.java:1646)

the error message indicate problem native memory, is, memory outside heap. garbage collector not responsible off-the-heap memory, why can not affect error garbage collector settings. excessive garbage collecting should not lead native memory issues (unless there bug in gc, of course).

what causing particular error hard without more information. assume have @ least 12 gb of available ram on machine? otherwise running heap big available memory.

some examples of things in java can exhaust native memory:

• leaks in native (jni) code

• direct allocation of many bytebuffers

or maybe there no leak, behavior expected application, , need allow larger processes or buy more memory.

first step troublehsoot check process size against possible process size limits. run such limit without there being actual leak.

if not problem, track process size on time, in relation available memory see if have "leaky" behavior.

if think have native memory leak, identify libraries use jni using (e.g. jdbc drivers) , try replace them java-only versions. also, check known memory leak bugs in such libraries , jvm version , websphere version.

if fails have resort native memory leak troubleshooting tools. question.