Some elasticsearch nodes not showing in Kibana / Health yellow

[karan_budhraja]

Hello.

I have an elasticsearch cluster of 7 nodes, however, only 5 are shown in Kibana monitoring with information (status, cpu usage, load average etc.). However, the number next to "Nodes" at the top left is 7. The 5 nodes that are visible use elasticsearch 6.4.0 and the 2 new ones use 6.7.1. They were being shown in Kibana before restarting the elasticsearch service on the servers, which perhaps points to a temporary elasticsearch setting which got wiped for those two nodes.

The nodes were visible on Kibana before restarting elasticsearch, so it shouldn't be a version mismatch problem. The 5 "good" nodes all have metricbeat running while the 2 problematic ones don't seem to have that installed (I could not find an installation). Assuming that the installation does not exist, that should not be the cause of the problem either since Kibana was showing these 2 nodes earlier.

The elasticsearch logs on the 2 problematic nodes keep saying the following:

`[2019-04-12T23:25:05,170][INFO ][o.e.x.m.e.l.LocalExporter] [myservername] waiting for elected master node [{oneofthe5goodservers}{somecode}{someothercode}{oneofthe5goodservers}{xx.xxx.xx.xxx:9300}{ml.machine_memory=somenumber, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)`

Nothing has changed with the master node or the KIbana node, so this x-pack issue must be on the side of the 2 problematic nodes. I tried adding the following lines to elasticsearch.yml to no avail (on those 2 nodes).

xpack.monitoring.collection.enabled: true
xpack.monitoring.elasticsearch.collection.enabled: true

Additionally, while Kibana does detect that 7 nodes exist in the cluster, the status is yellow and I'm not sure why. Perhaps it is related to the problem pointed out in the logs. Using commands like

GET /_nodes/stats
GET _cat/nodes?v
GET /_cluster/settings?include_defaults=true

indicate that all 7 elasticsearch nodes are recognized and functional.

Please let me know if you have any suggestions on what dynamic setting may have gotten lost and how to fix it, or where this problem points to in general.

Thanks!

[Mike_Place]

@karan_budhraja

to setup local exporter

Hi Karan and welcome to the Elastic forums.

Were the two nodes which are not being fully recognized previously 6.4.0 nodes and were then upgraded or were they fresh installs of 6.7.1?

Best,
-Mike

[shaunak (Shaunak Kashyap)]

Hi Karan,

In addition to @Mike_Place's question above, I have a few more questions as well:

1. could you post the output of GET _cat/indices?v please?

2. could you post the output of GET _cat/nodes?v

3. could you post the output of GET _cluster/settings?include_defaults&filter_path=*.xpack.monitoring.*

4. could you post all xpack.monitoring.* settings from one of your "good" node's elasticsearch.yml? Similarly, could you post all xpack.monitoring.* settings from one of your "bad" node's elasticsearch.yml?

If there's any sensitive information in any of the outputs above, please mask it before posting here.

Thanks,

Shaunak

[shaunak (Shaunak Kashyap)]

Sorry Karan, one more request. Could you please post the output of the following query as well?

POST .monitoring-es-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": "node_stats"
          }
        },
        {
          "range": {
            "timestamp": {
              "format": "epoch_millis",
              "gte": "now-1m/m",
              "lte": "now/m"
            }
          }
        }
      ]
    }
  },
  "collapse": {
    "field": "source_node.uuid"
  },
  "sort": [
    {
      "timestamp": {
        "order": "desc"
      }
    }
  ]
}

Thanks,

Shaunak

[karan_budhraja]

Hi Mike.

They were fresh installs of 6.7.1.

Thanks,
Karan

[Mike_Place]

Thanks for the reply, @karan_budhraja. When you have the information that @shaunak requested then we can proceed. Cheers!