Hello.
I have an elasticsearch cluster of 7 nodes, however, only 5 are shown in Kibana monitoring with information (status, cpu usage, load average etc.). However, the number next to "Nodes" at the top left is 7. The 5 nodes that are visible use elasticsearch 6.4.0 and the 2 new ones use 6.7.1. They were being shown in Kibana before restarting the elasticsearch service on the servers, which perhaps points to a temporary elasticsearch setting which got wiped for those two nodes.
The nodes were visible on Kibana before restarting elasticsearch, so it shouldn't be a version mismatch problem. The 5 "good" nodes all have metricbeat running while the 2 problematic ones don't seem to have that installed (I could not find an installation). Assuming that the installation does not exist, that should not be the cause of the problem either since Kibana was showing these 2 nodes earlier.
The elasticsearch logs on the 2 problematic nodes keep saying the following:
`[2019-04-12T23:25:05,170][INFO ][o.e.x.m.e.l.LocalExporter] [myservername] waiting for elected master node [{oneofthe5goodservers}{somecode}{someothercode}{oneofthe5goodservers}{xx.xxx.xx.xxx:9300}{ml.machine_memory=somenumber, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)`
Nothing has changed with the master node or the KIbana node, so this x-pack issue must be on the side of the 2 problematic nodes. I tried adding the following lines to elasticsearch.yml to no avail (on those 2 nodes).
xpack.monitoring.collection.enabled: true
xpack.monitoring.elasticsearch.collection.enabled: true
Additionally, while Kibana does detect that 7 nodes exist in the cluster, the status is yellow and I'm not sure why. Perhaps it is related to the problem pointed out in the logs. Using commands like
GET /_nodes/stats
GET _cat/nodes?v
GET /_cluster/settings?include_defaults=true
indicate that all 7 elasticsearch nodes are recognized and functional.
Please let me know if you have any suggestions on what dynamic setting may have gotten lost and how to fix it, or where this problem points to in general.
Thanks!