I have looked at many posts in the discussion forum and googled quite a bit as well. Either these posts are for older versions or there wasn't anything that matched the issue I am seeing. Hence, posting it one more time. Apologies if there is something really obvious that I am overlooking.
There are a 5 data nodes in our cluster and data seem to be distributed evenly, but only one or two data nodes consistently seem to have high heap usage (between 85-90%). Can someone help me out with troubleshooting steps? Few things I have looked at:
- Cluster health / unassigned shards : 0
- Number of pending tasks: 0
- Script stats: look normal
- Field data: between 10-15% of memory
Data Node configuration: 16 cores, 30 GB RAM, 15 GB for lucene and ES each
ES version: 5.3.2
I couldn't interpret anything meaningful from other stats. Please see node stats in my onedrive
We don't have X-Pack and installing one in production is out of scope as of now. Let me know if you need more info.