Previously, We were using single node for our log monitoring purpose. But then due to heavy read and write we have to switch to cluster. So our cluster has 2 nodes both as mastter and data, and 1 node as cache only. We're using Elasticsearch 5.3.0. So now our kibana make request to cache only node to cache requests, and our logstash includes both other nodes as hosts so that if any 1 one goes down the other can handle the input.
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.31.62.213 54 71 0 0.00 0.01 0.05 - - coordinate-node-1
172.31.52.40 46 88 2 0.02 0.04 0.05 md * elasticsearch-node-2
172.31.62.202 76 98 42 1.35 1.26 1.36 md - elasticsearch-node-1
But I'm unable to understand why my new indices health is not green. Every index has 5 shards and 1 replica as default. Notable thing is some indices have their health status green but other don't. Please help. Is my strategy wrong or am i supposed to do something manually ?