GC problems in ES 1.4.4

Elastic Stack Elasticsearch
1

Frequent full GC even if the memory utilization is 30-40%. We have not changed the default JVM settings

Cluster details -
a. One master node, 2 data nodes(15 GB Heap size)
b. All the replicas are allocated on one node and all the primaries are on the other node. It should not be a problem, right? Also frequent full GC is observed on one of the 2 data nodes.
c. Shards - 75 shards with replication as 1. I guess the problem is because of shard over allocation. Most of the queries we hit are aggregation queries.
d. mlockall is set to true

Thanks, Rushiraj

2

What do the GC logs show?

3

GC logs snippet -
493999.583: [Full GC493999.583: [CMS: 1308461K->1310104K(15047104K), 2.6996450 secs] 1534368K->1310104K(15660544K), [CMS Perm : 83967K->83967K(83968K)], 2.6998090 secs] [Times: user=2.75 sys=0.00, real=2.70 secs]

As you can see, full gc is getting triggered for heap utilization as low as 1308461K. Total heap size is 15047104K. This is happening on one of the two data nodes.

4

That's an unusual GC log event, is that from ES?

5

Yes. It is from ES. The problem is with perm gen memory utilization. I guess that is triggering the full gc.