Elastic Search 2.2 Occupying a lot of ram which leads to GC Memory Pressure

Elastic Stack Elasticsearch
1

Hi,
We are indexing around 1000 logs per second with the elasticsearch in Client mode. But as soon as we start the product the elasticsearch occupies around 1 GB of the total heap. This however grows > 4 gb in a few mins. We took a heap dump and found the leak suspect show the following.

969,075,752 bytes (45.73 %) of Java heap is used by 39 instances of org/elasticsearch/indices/IndicesService$OldShardsStats
Contains 25 instances of the following leak suspects:

  • array of org/apache/lucene/index/IndexReaderContext holding 32,745,720 bytes at 0x593493920
  • array of org/apache/lucene/index/IndexReaderContext holding 31,193,464 bytes at 0x564a167b8
  • array of org/apache/lucene/index/SegmentReader holding 41,076,960 bytes at 0x564a1bd18
  • array of org/apache/lucene/index/SegmentReader holding 42,197,240 bytes at 0x57a00e140
  • array of org/apache/lucene/index/SegmentReader holding 55,882,368 bytes at 0x57f847e78
  • array of org/apache/lucene/index/SegmentReader holding 51,970,776 bytes at 0x5ae887958
  • array of org/apache/lucene/index/IndexReaderContext holding 37,529,976 bytes at 0x544983b30
  • array of org/apache/lucene/index/IndexReaderContext holding 23,545,392 bytes at 0x576e6a478
  • array of org/apache/lucene/index/IndexReaderContext holding 24,504,016 bytes at 0x5862190f8
  • array of org/apache/lucene/index/SegmentReader holding 45,007,792 bytes at 0x54529dfb0
  • array of org/apache/lucene/index/SegmentReader holding 31,010,832 bytes at 0x561789018
  • array of org/apache/lucene/index/SegmentReader holding 20,942,600 bytes at 0x5ab95c6f0
  • array of org/apache/lucene/index/IndexReaderContext holding 34,278,344 bytes at 0x5ae887550
  • array of org/apache/lucene/index/SegmentReader holding 50,109,752 bytes at 0x5624ab938
  • array of org/apache/lucene/index/IndexReaderContext holding 16,458,112 bytes at 0x54673d7f8
  • array of org/apache/lucene/index/SegmentReader holding 31,903,696 bytes at 0x592cf0b68
  • array of org/apache/lucene/index/IndexReaderContext holding 29,125,936 bytes at 0x544fc4318
  • array of org/apache/lucene/index/IndexReaderContext holding 45,398,144 bytes at 0x54c206ed8
  • array of org/apache/lucene/index/SegmentReader holding 15,560,120 bytes at 0x5ae7b9930
  • array of org/apache/lucene/index/IndexReaderContext holding 30,677,160 bytes at 0x55d3b3f48
  • array of org/apache/lucene/index/IndexReaderContext holding 39,807,240 bytes at 0x55ec255e8
  • array of org/apache/lucene/index/SegmentReader holding 52,102,584 bytes at 0x5811e7958
  • array of org/apache/lucene/index/SegmentReader holding 54,997,248 bytes at 0x58215e460
  • array of org/apache/lucene/index/IndexReaderContext holding 30,658,880 bytes at 0x57a052b88
  • array of org/apache/lucene/index/SegmentReader holding 34,094,208 bytes at 0x57a44bbf8
2

Elasticsearch 2.2 is very old and reached the end of its supported life about 18 months ago. More recent versions have seen improvements in how they handle excess load and how well they work with small heaps. Can you reproduce this with a more recent version?

3

Since our issue is in production. It is would take time to change it to the latest version . So is there any workaround to control the heap growth in the old versions

4

It's hard to say without further investigation, which is hard to do on such an old version.

Did this problem only start recently?

5

no we have this problem on most of our servers when the product is running for too long. This also causes the GC to run more than a couple of minutes.

Took another heap dump and found the same issue of memory usage with ES.

Problem Suspect 1

One instance of "org.elasticsearch.transport.TransportService" loaded by "sun.misc.Launcher$AppClassLoader @ 0xf7c7fde8" occupies 665,320,368 (10.34%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Node[]" loaded by "<system class loader>" .

Keywords
java.util.concurrent.ConcurrentHashMap$Node
org.elasticsearch.transport.TransportService
sun.misc.Launcher$AppClassLoader @ 0xf7c7fde8

73 instances of "org.elasticsearch.index.shard.IndexShard" , loaded by "sun.misc.Launcher$AppClassLoader @ 0xf7c7fde8" occupy 3,142,511,760 (48.82%) bytes.

Biggest instances:

  • org.elasticsearch.index.shard.IndexShard @ 0x119f288a8 - 162,221,880 (2.52%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x195c39da0 - 152,507,792 (2.37%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x178fa6748 - 142,285,616 (2.21%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x17ada5a48 - 136,175,728 (2.12%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x17bf60568 - 132,623,744 (2.06%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x14b4e5000 - 132,606,224 (2.06%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x13497d7a0 - 130,126,800 (2.02%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x157539aa0 - 128,982,624 (2.00%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x15936f5c8 - 125,072,992 (1.94%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1bd4dc5f8 - 119,272,712 (1.85%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1301bf868 - 116,931,232 (1.82%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x12e4a3c68 - 116,545,608 (1.81%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x15fefd508 - 116,302,392 (1.81%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x188069be0 - 114,275,768 (1.78%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1b99871b8 - 111,497,552 (1.73%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1185927e0 - 108,781,488 (1.69%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1a2d328e8 - 105,745,416 (1.64%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1bec588c0 - 101,327,920 (1.57%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1255938b0 - 95,205,768 (1.48%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x13cc428b0 - 94,875,752 (1.47%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1587e58c8 - 87,836,288 (1.36%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x13497d5a0 - 84,969,368 (1.32%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x1557f88a0 - 80,969,376 (1.26%) bytes.* org.elasticsearch.index.shard.IndexShard @ 0x175085620 - 80,708,144 (1.25%) bytes.

These instances are referenced from one instance of "java.util.concurrent.RunnableScheduledFuture[]" , loaded by "<system class loader>"

Keywords
org.elasticsearch.index.shard.IndexShard
sun.misc.Launcher$AppClassLoader @ 0xf7c7fde8
java.util.concurrent.RunnableScheduledFuture

6

This report only accounts for ~3GB of heap. How much heap does this node have?

7

This is a leak report apart from this es has ~1.5 GB on a 6GB heap. So occupies around 4.5 GB of it

8

6GB isn't really a lot of heap. Can you tell us a bit more about this cluster? How many shards does it have, and how many nodes?

9

1 shard and 1 node. Also we are facing high GC STW ~ 2-5 mins

10

The summary you posted above shows at least 24 shards, so I don't understand how this cluster only has one shard.

11

Capture
Capture.PNG829×459 22.1 KB

This is our config and has only one shard configured

12

I think that sets each new index to have 1 shard, unless you override this when creating the index, and doesn't tell us how many shards there are in total. How many shards does GET _cat/shards list?

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.