All Data nodes are offline except master node in Stack monitoring

[Sagar_Mandal]

Hi Team,

So my cluster with one master node and two data node is up and running but when i check in stack monitoring in kibana, only data nodes are offline and master is up.

but when i check indices in each node they are all same.
I'm able to get data and everything but this is only in stack monitoring

really need help on this,
any possible solution would help me.

Thanks and regards,
Sagar Mandal

[rashmi (kulkarni)]

I'm not seeing anything obvious to me.

What about the elasticsearch.yml for the data nodes? And just to double check, I'm assuming you are not seeing any errors in the data nodes server logs?

It looks like the other two nodes are not reporting node_stats at all. Let's see if they are reporting any monitoring data.

Try running:

POST .monitoring-es-*/_search
{
  "size": 1,
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ],
  "query": {
    "term": {
      "type": {
        "value": "shards"
      }
    }
  },
  "collapse": {
    "field": "shard.node"
  }
}

Check if the only node you see is also the only one showing up in the UI.

If so, we need to figure out why those nodes aren't reporting.

A couple of things that will help figure that out:

1. GET _cluster/settings for the cluster
2. The elasticsearch.yml for each node

hope it helps
Thanks
Rashmi