GC problems in ES 1.4.4


(Rushiraj Chavan) #1

Frequent full GC even if the memory utilization is 30-40%. We have not changed the default JVM settings

Cluster details -
a. One master node, 2 data nodes(15 GB Heap size)
b. All the replicas are allocated on one node and all the primaries are on the other node. It should not be a problem, right? Also frequent full GC is observed on one of the 2 data nodes.
c. Shards - 75 shards with replication as 1. I guess the problem is because of shard over allocation. Most of the queries we hit are aggregation queries.
d. mlockall is set to true

Thanks, Rushiraj


(Mark Walkom) #2

What do the GC logs show?


(Rushiraj Chavan) #3

GC logs snippet -
493999.583: [Full GC493999.583: [CMS: 1308461K->1310104K(15047104K), 2.6996450 secs] 1534368K->1310104K(15660544K), [CMS Perm : 83967K->83967K(83968K)], 2.6998090 secs] [Times: user=2.75 sys=0.00, real=2.70 secs]

As you can see, full gc is getting triggered for heap utilization as low as 1308461K. Total heap size is 15047104K. This is happening on one of the two data nodes.


(Mark Walkom) #4

That's an unusual GC log event, is that from ES?


(Rushiraj Chavan) #5

Yes. It is from ES. The problem is with perm gen memory utilization. I guess that is triggering the full gc.


(system) #6