I recently upgraded 3 (of 9) nodes to new hardware. Since that time, one of the nodes (es02) is not visible in the Kibana "Monitoring" page. ALL nodes are visible via Elasticsearch Head plugin.
Version 6.8.3
Based on earlier report from another user (aparently abandoned by the user), I will attach the results requested by one of the ES developers for that ticket.
In the results below, the nodes are es01,02,03,5,06,07,08,09, and hd3. The node that is not displaying is es02; the other new machines that are configured the same way and are working fine are es01 and es03. Please note that the node appears to be otherwise working fine.
GET _cat/indices?v
File: indices.txt
GET _cat/nodes?v
File: nodes.json
GET _cluster/settings?include_defaults&filter_path=.xpack.monitoring.
File: cluster.json
POST .monitoring-es-*/_search
File: monitoring.json
Could you post all xpack.monitoring.* settings from one of your "good" node's elasticsearch.yml? Similarly, could you post all xpack.monitoring.* settings from one of your "bad" node's elasticsearch.yml?
[root@es02:/etc/elasticsearch]# grep xpack elasticsearch.yml #xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true #xpack.monitoring.elasticsearch.collection.enabled: true
(The commented out lines were tried based on suggestions found elsewhere)
** Please advise how/where to send the result files. I thought I could attach them here.
Could you please query the cluster where the monitoring data is stored with the following?
GET /.monitoring-es*/_search?filter_path=hits.hits._source.node_stats.node_id
You should get a list of node IDs. Could you please compare that list to what's returned by GET /_nodes?filter_path=nodes.*.name see if the node that's not appearing (es02) is present?
@thealy Are you certain that GET /.monitoring-es*/_search?filter_path=hits.hits._source.node_stats.node_id was run from the monitoring cluster? The other nodes should be shown there, so this is a bit surprising.
Sorry for the delay here. In looking at this gist, it does appear that @chrisronline is on the right track. I see 8 of the 9 machines in this list and es02 is indeed missing. That suggests that although the data may be indexed, it may have an incorrect time stamp. Might it be the case that es02 is somehow an outlier with the timestamps it is setting?
We are also facing similar problem. Our cluster comprises of 3 master, 3 coordinate, 4 hot and 4 warm on 6.7.1 version. We upgrade one master and one warm node to 6.8.3 but those are not visible in Kibana monitoring console.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.