Heap Usage is not as usual

[Nicolas_Garfinkiel]

Hello all,

I have a 3 nodes cluster with two main indices, 3 shards and 3 replicas each. Each node has total 16GB RAM with 6GB reserved to ES. I see the following pattern.

Memory percent starts at ~14% and in 22/24hs it reaches ~75%. That's when old collection GC kicks in and memory goes down at ~14%.

Suddenly two days ago, one of the nodes (master node) reached ~75% in 12hs instead of the usual 22/24hs. Performance is still superb and the node is super responsive, but I'm just curious as to why this happened. I don't see anything special in the logs, besides scattered infos about GC overhead and seldom any WARN because it reached 500ms+. That's for all nodes.

What I noticed is that usually old collection adds 2/3mb per minute per node, but in this node I see sudden spikes of hundreds of MBs, which makes it deviate a lot from what was so far the standard behavior.

Still performance is great and old collection GC is in the ~300ms time, so it's not a big a deal, but I just want to make sure everything is fine now that indices sizes are not that big (8GB total, 3M docs).

Thanks a lot in advance!

[warkolm (Mark Walkom)]

What version?

[Nicolas_Garfinkiel]

Oh, so sorry! ES 5.3.2, Oracle JDK 1.8u55 running on Windows Server 2012 R2.

[warkolm (Mark Walkom)]

Given the data size, it'll depend on what sort of data it is, the mappings and the queries you are running.

[Nicolas_Garfinkiel]

Yes, of course. But what's weird is that: other than the node being in question is the master node, all three nodes are the same and the other two didn't present this behavior. That's what puzzles me. All of them have the same information, because they have it's primary shard and two replicas from the other two nodes.

[Nicolas_Garfinkiel]

Oh, and as to usage. Mappings have one nested field with roughly ~5 items average, almost all queries are constant_score with a filter. I use a lot of sorting and sometimes I retrieve thousands of docs. Another interesting fact is that docs are inserted and then updated several times. So I have way more Lucene docs (with deletes) than actual docs.