My problem turned out to be mis-match of time setting on the system that was not showing. Try running ntpupdate or install ntp if not already.
As per the output of the first query, node id for upgraded master is "m_7bmIwbRoi-j80tDms2lw" and node id for upgraded warm node is "a5_1IeuxSji6QYlo3SHQ6Q".
Output of query GET /_nodes?filter_path=nodes.*.name is
{
"nodes" : {
"m_7bmIwbRoi-j80tDms2lw" : {
"name" : "master-1"
},
"A8Qe1_q3QUqYPsgw3jcz5g" : {
"name" : "hot-2"
},
"gdoeR2EzTvCuE4RDeeb_aA" : {
"name" : "hot-3"
},
"XZFg8VpuRS2WIse9KBIvMA" : {
"name" : "coordinate-1"
},
"yvOSip5mQRG3uThwezXGBg" : {
"name" : "WARM-2"
},
"JNSXpU_1SP-nYn5lxWxZXQ" : {
"name" : "hot-4"
},
"ebvjKCrhQvK8581Uvr8Gew" : {
"name" : "master-2"
},
"GdXTW8qmR2aDXclXrQAgRw" : {
"name" : "coordinate-3"
},
"Qfpu2zrJTHaP_ngUNLZ7Jw" : {
"name" : "hot-1"
},
"xDtX5E1JRnu2BNv8zHe8bQ" : {
"name" : "coordinate-2"
},
"djCtwvYuSgWcgQ2DnbvitQ" : {
"name" : "master-3"
},
"2OazDzlySLWXeTlNlkQ-cA" : {
"name" : "WARM-2"
},
"fz14QsFFRYijYJqOWRVONw" : {
"name" : "warm-1"
},
"XYIU3i3cRw-C7jAdDRj17g" : {
"name" : "WARM-4"
}
}
}
The upgraded nodes are master-1 and warm-1. Output of above query and GET /_nodes/stats show node id for warm-1 to be "fz14QsFFRYijYJqOWRVONw" whereas output of
GET .monitoring-es-*/_search?filter_path=aggregations.node_ids
{
"size": 0,
"query": {
"term": {
"type": {
"value": "node_stats"
}
}
},
"aggs": {
"node_ids": {
"terms": {
"field": "node_stats.node_id",
"size": 50
}
}
}
}
doesn't contain "fz14QsFFRYijYJqOWRVONw". Instead it contains "a5_1IeuxSji6QYlo3SHQ6Q" as one of the node ids.
I found following logs in the upgraded master node:
[o.e.x.m.e.l.LocalExporter] [master-1] monitoring index template [.monitoring-logstash] does not exist, so service cannot start (waiting on master)
[o.e.x.m.e.l.LocalExporter] [master-1] waiting for elected master node [{master-3}{djCtwvYuSgWcgQ2DnbvitQ}{sdvoobvCS6ig6brEOUmMPA}{<ip>}{<ip>:9300}{xpack.installed=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[o.e.x.m.e.Exporters ] [master-1] skipping exporter [default_local] as it is not ready yet
The last one keeps repeating every 10 seconds.
X-pack monitoring is enabled on all the nodes.
I saw below issues and it seems like I'm facing problems due to different versions of elasticsearch.
skipping exporter [default_local] as it is not ready yet
Rebuilt node, logs say waiting for local exporter
Rolling upgrade sequence of node types
Will the issue be resolved if I upgrade all the nodes to 6.8.3?
Is there anything in the logs on this node indicating an issue? That's the elected master and expected to create the necessary prerequisites for monitoring to work properly (in this case, index templates)
It might be worth seeing what templates exist and which don't, but running:
GET _template/.monitoring*
I found following in the elected master
[o.e.c.s.ClusterApplierService] [master-3] processing [apply cluster state (from master [master {master-3}{djCtwvYuSgWcgQ2DnbvitQ}{sdvoobvCS6ig6brEOUmMPA}{<ip>}{<ip>:9300}{xpack.installed=true} committed version [91269] source [zen-disco-node-join[{master-1}{m_7bmIwbRoi-j80tDms2lw}{XsBFY4ahTC-ixXbNdeTW-w}{<ip>}{<ip>:9300}{xpack.installed=true}]]])]: took [25ms] done applying updated cluster state (version: 91269, uuid: gsZ5opT3R3uTcaJ_mpVChA)
[o.e.c.s.MasterService ] [master-3] processing [zen-disco-node-join[{master-1}{m_7bmIwbRoi-j80tDms2lw}{XsBFY4ahTC-ixXbNdeTW-w}{<ip>}{<ip>:9300}{xpack.installed=true}]]: took [4.8s] done publishing updated cluster state (version: 91269, uuid: gsZ5opT3R3uTcaJ_mpVChA)
[o.e.c.s.MasterService ] [master-3] processing [master ping (from: {master-1}{m_7bmIwbRoi-j80tDms2lw}{XsBFY4ahTC-ixXbNdeTW-w}{<ip>}{<ip>:9300}{xpack.installed=true})]: execute
[o.e.c.s.MasterService ] [master-3] processing [master ping (from: {master-1}{m_7bmIwbRoi-j80tDms2lw}{XsBFY4ahTC-ixXbNdeTW-w}{<ip>}{<ip>:9300}{xpack.installed=true})]: took [0s] no change in cluster state
[o.e.x.m.e.l.LocalExporter] [master-3] monitoring index templates and pipelines are installed on master node, service can start
I did not find any error in the logs except the following:
[c.f.s.r.IndexResolverReplacer] [master-3] class org.elasticsearch.action.admin.indices.template.put.PutIndexTemplateRequest not supported (It is likely not a indices related request)
Since, we found above logs in the upgraded master, I am posting the output of GET _template/.monitoring-logstash
I have upgraded all the nodes to 6.8.3 and now kibana monitoring is showing all the nodes. I'll upgrade kibana to 6.8.3.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.