java - Garbage collector not able to clear garbage, causes full frequent gc -

we use parallel gc 1.7.0_71.

java -xx:+printcommandlineflags -xx:+printgcdetails -version -xx:initialheapsize=258222272 -xx:maxheapsize=4131556352 -xx:+printcommandlineflags -xx:+printgcdetails -xx:+usecompressedoops -xx:+useparallelgc  java version "1.7.0_71" java(tm) se runtime environment (build 1.7.0_71-b14) java hotspot(tm) 64-bit server vm (build 24.71-b01, mixed mode) 

we seeing huge garbage created in our application every few minutes under decent load . minor gc triggers every few seconds , major gc triggers every 2 minutes. on analysis found 1.8 million char array objects each of size 173k left in heap. minor gc not able recover them . on taking heapdump ,we found lot of objects in remainder section in eclipe mat. mat histogram shows lots of char arrays (they rendered html's),however incoming references , merge paths gc root char array unreachable, full gc able recover them not minor gc. why minor gc not able recover them when there no referenced objects ? eclipse mat pictures attached. please see references unreachable.

there reference request objects, cleared them setting nulls.

before fix there hanging references httpservletrequest objects

images related issue: https://www.dropbox.com/sh/qgsitzb7x27j8kc/aaboqwr1qpwtpidto6b0_pm7a?dl=0

overall heap

heap histogram view question

1. do think issue garbage collector? upgraded jboss 7.1.1 wildfly 8.2.0.final. no changes in gc strategy , jdk version. why gc not able reclaim memory pointing unreachable references.

2. we can decrease xx:newratio 1. no sure work full gc happens

3. do think moving g1 gc ? there decrease in throughput ? best options available g1 gc. our heap size -xms : 1024m , -xmx 2048m. perm gen 512m not seeing memory leaks, no out of memory error.attaching full gc log outputs

   2015-05-10 19:32:41 ist| 459.939: [full gc [psyounggen: 8123k->0k(680960k)] [paroldgen: 782136k->359065k(766464k)] 790260k->359065k(1447424k) [pspermgen: 202932k->202930k(441344k)], 1.0738240 secs] [times: user=3.37 sys=0.01, real=1.07 secs] 2015-05-10 19:32:42 ist| 462.306: [gc [psyounggen: 672768k->10534k(685056k)] 1031833k->369600k(1451520k), 0.0450800 secs] [times: user=0.15 sys=0.00, real=0.04 secs] 2015-05-10 19:32:44 ist| 463.641: [gc [psyounggen: 682790k->9093k(685568k)] 1041856k->373085k(1452032k), 0.0570820 secs] [times: user=0.16 sys=0.00, real=0.06 secs] 2015-05-10 19:32:45 ist| 464.936: [gc [psyounggen: 681349k->9812k(686080k)] 1045341k->377511k(1452544k), 0.0439060 secs] [times: user=0.12 sys=0.00, real=0.04 secs] 2015-05-10 19:32:46 ist| 466.283: [gc [psyounggen: 683092k->10733k(686080k)] 1050791k->383554k(1452544k), 0.0464700 secs] [times: user=0.14 sys=0.00, real=0.05 secs] 2015-05-10 19:32:48 ist| 467.659: [gc [psyounggen: 684013k->11283k(685568k)] 1056834k->388651k(1452032k), 0.1381130 secs] [times: user=0.30 sys=0.00, real=0.14 secs] 2015-05-10 19:32:50 ist| 469.734: [gc [psyounggen: 684051k->9652k(686080k)] 1061419k->393759k(1452544k), 0.0466800 secs] [times: user=0.13 sys=0.00, real=0.05 secs] 2015-05-10 19:32:51 ist| 471.087: [gc [psyounggen: 682420k->11253k(685568k)] 1066527k->400087k(1452032k), 0.0589180 secs] [times: user=0.11 sys=0.00, real=0.06 secs] 2015-05-10 19:32:52 ist| 472.325: [gc [psyounggen: 684021k->7957k(686080k)] 1072855k->403018k(1452544k), 0.0436140 secs] [times: user=0.13 sys=0.00, real=0.04 secs] 2015-05-10 19:32:54 ist| 473.606: [gc [psyounggen: 680725k->9177k(685056k)] 1075786k->406493k(1451520k), 0.0524990 secs] [times: user=0.13 sys=0.00, real=0.05 secs]  2015-05-10 19:34:34 ist| 573.526: [gc [psyounggen: 684217k->10956k(686080k)] 1440629k->771626k(1452544k), 0.0416620 secs] [times:  user=0.14 sys=0.00, real=0.04 secs]  2015-05-10 19:34:34 ist| 573.568: [full gc [psyounggen: 10956k->0k(686080k)] [paroldgen: 760670k->364958k(818688k)] 771626k->364958k(1504768k) [pspermgen: 203069k->203069k(420864k)], 0.8001740 secs] >[times: user=2.46 sys=0.01, real=0.80 secs] 2015-05-10 19:34:36 ist| 575.600: [gc [psyounggen: 674304k->10465k(686592k)] 1039262k->375423k(1505280k), 0.0410330 secs] [times: user=0.13 sys=0.00, real=0.04 secs] 2015-05-10 19:36:35 ist| 694.277: [gc [psyounggen: 684413k->9342k(687104k)] 1490469k->820033k(1505792k), 0.2160320 secs] [times: user=0.55 sys=0.08, real=0.21 secs] 2015-05-10 19:36:36 ist| 695.664: [gc [psyounggen: 684670k->8323k(687104k)] 1495361k->823380k(1505792k), 0.0454050 secs] [times: user=0.11 sys=0.00, real=0.05 secs] 2015-05-10 19:36:37 ist| 695.710: [full gc [psyounggen: 8323k->0k(687104k)] [paroldgen: 815056k->363295k(838144k)] 823380k->363295k(1525248k) [pspermgen: 203095k->203095k(401920k)], 0.8133080 secs] [times: user=2.43 sys=0.01, real=0.81 secs] 2015-05-10 19:36:38 ist| 697.669: [gc [psyounggen: 675328k->10586k(686592k)] 1038623k->373882k(1524736k), 0.0436000 secs] [times: user=0.13 sys=0.00, real=0.04 secs] ... .... 

why minor gc not able recover them when there no referenced objects ?

most because in old generation. minor gcs dispose of unreachable objects in young generation.

this happens when object lives longer can remain in young generation due tenuring. e.g. if caches involved hold onto results while or when request lifetime exceeds of gc interval * tenuring threshold.

-xx:+printtenuringdistribution may prove informative.

first try providing pause time goal via -xx:maxgcpausemillis=.... parallelgc might able meet it.

if doesn't refactor code reduce object lifetimes or reduce allocation rates make minor gcs less frequent.

note foremost parallelgc throughput collector, in terms of cpu cycles it's more efficient concurrent collectors won't able meet low pause time goals those.

do think moving g1 gc ?

if concerned pause times, likely. might want try cms.

if want try out g1 should switch java 8, heuristics have improved lot on time , still maturing.

will there decrease in throughput ?

possibly. depends on whether there's spare cpu capacity , how define/measure throughput. , under less favorable circumstances decrease might not significant.

what best options available g1 gc.

g1 supposed self-tuning (beyond user-provided pause , throughput goals apply parallelgc). enable , see if provides acceptable performance.