I have a 5 node cluster on kubernetes, 2 data nodes, 3 masters. This is sitting underneath graylog. For some reason, usually only 1 data node is extremely busy. And lately the load really high on it. I've grabbed the hot threads and pasted in link below. The specs are as follows for data nodes:
6 CPU cores, 16GB ram, 12GB heap - 1.15TB of total disk - 526GB available on each.
any suggestions on optimizing? this is running on es 6.8.3, JVM 12.0.2