When upgrading from any version to a higher version that is supported by Elastic, what is the order of upgrade for node types: master, index-only, coordination-only ?
I'd read the upgrade guide.
This links also to the specific elasticsearch upgrade documentation.
I don't think there is any specific order but I'd probably start by the dedicated master nodes if you have that type of nodes.
@dadoonet I upgraded a coordination node first. My master nodes have x-pack license applied. However, I am still getting the following error.
[2019-09-08T23:51:04,568][INFO ][o.e.x.m.e.l.LocalExporter] [elkcoord02.mycluster] waiting for elected master node [{elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
That's not an error message, just INFO, and in fact there's an open issue to remove them to avoid this confusion. These messages should stop once you've completed your upgrade.
1 Like
@DavidTurner I initially thought of ignoring the INFO message as well. However, I saw that the upgraded coordination node is unable to join the cluster. In monitoring app of Kibana, I find the other coordination nodes that are at previous version but not the upgraded one.
I am confused. The log message you quote indicates that it has successfully joined the cluster.
I suspect that the message indicates successful joining to the cluster. I restarted the node and the logs are as follows:
[2019-09-09T10:00:12,284][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-deprecation]
[2019-09-09T10:00:12,284][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-graph]
[2019-09-09T10:00:12,284][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-ilm]
[2019-09-09T10:00:12,284][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-logstash]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-ml]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-monitoring]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-rollup]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-security]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-sql]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-upgrade]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded module [x-pack-watcher]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService ] [elkcoord02.mycluster] loaded plugin [mapper-size]
[2019-09-09T10:00:16,843][INFO ][o.e.x.s.a.l.LdapUserSearchSessionFactory] [elkcoord02.mycluster] Realm [myrealm] is in user-search mode - base_dn=[my_dn], search filter=[(uid={0})]
[2019-09-09T10:00:17,164][INFO ][o.e.x.s.a.s.FileRolesStore] [elkcoord02.mycluster] parsed [10] roles from file [/etc/elasticsearch/roles.yml]
[2019-09-09T10:00:17,835][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elkcoord02.mycluster] [controller/120607] [Main.cc@109] controller (64 bit): Version 6.8.2 (Build f4a9b28c0d5114) Copyright (c) 2019 Elasticsearch BV
[2019-09-09T10:00:18,485][DEBUG][o.e.a.ActionModule ] [elkcoord02.mycluster] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-09-09T10:00:18,560][INFO ][o.e.d.DiscoveryModule ] [elkcoord02.mycluster] using discovery type [zen] and host providers [settings]
[2019-09-09T10:00:19,482][INFO ][o.e.n.Node ] [elkcoord02.mycluster] initialized
[2019-09-09T10:00:19,482][INFO ][o.e.n.Node ] [elkcoord02.mycluster] starting ...
[2019-09-09T10:00:19,654][INFO ][o.e.t.TransportService ] [elkcoord02.mycluster] publish_address {10.240.1.55:9401}, bound_addresses {[::]:9401}
[2019-09-09T10:00:19,667][INFO ][o.e.b.BootstrapChecks ] [elkcoord02.mycluster] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-09-09T10:00:23,591][INFO ][o.e.c.s.ClusterApplierService] [elkcoord02.mycluster] detected_master {elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, added {{elkdatahot02.mycluster}{II5K2KjbQ_OXvJgBPo6sgw}{KQS6FVOuTQmP3DUSD0Blxw}[snipped] committed version [6727]])
[2019-09-09T10:00:24,695][INFO ][o.e.x.s.a.TokenService ] [elkcoord02.mycluster] refresh keys
[2019-09-09T10:00:25,179][INFO ][o.e.x.s.a.TokenService ] [elkcoord02.mycluster] refreshed keys
[2019-09-09T10:00:25,192][INFO ][o.e.l.LicenseService ] [elkcoord02.mycluster] license [value] mode [snipped] - valid
[2019-09-09T10:00:25,222][INFO ][o.e.h.n.Netty4HttpServerTransport] [elkcoord02.mycluster] publish_address {10.240.1.55:9411}, bound_addresses {[::]:9411}
[2019-09-09T10:00:25,223][INFO ][o.e.n.Node ] [elkcoord02.mycluster] started
[2019-09-09T10:00:50,131][INFO ][o.e.x.m.e.l.LocalExporter] [elkcoord02.mycluster] waiting for elected master node [{elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
The node is still not part of the cluster. Am I missing something? Let me know if I can provide any other info.
@DavidTurner, @dadoonet As per the above logs, the master was detected but then why is the coordination node not added to the cluster?
Interestingly, we are only seeing an INFO log.
I am upgrading from 6.6.1 to 6.8.
What is making you think this? These logs indicate that this node joined the cluster. What does GET _cat/nodes report?
I can see from the monitoring dashboard that the node is not added.
However, GET _cat/nodes reports that the node is present. That is my source of confusion. Please suggest.
I see. Hmm. I don't know why it's not picked up on the monitoring dashboard, but unfortunately that's not my area of expertise.
Can you suggest someone who could answer? Should I open an issue on Github?
Perhaps start with a new post in the Kibana forum since the question is no longer really about upgrade order.
It could be because you selected a specific timeframe in Kibana instead the last 15 minutes for example.
@dadoonet I have checked changing the time window from last 1 hour to last 4 hours and last 12 hours but the result is the same.
So may be it's a bug. As @DavidTurner wrote, you should ask in #kibana forum may be...
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.