Upgrade order

Elastic Stack Elasticsearch
1

When upgrading from any version to a higher version that is supported by Elastic, what is the order of upgrade for node types: master, index-only, coordination-only ?

2

I'd read the upgrade guide.

This links also to the specific elasticsearch upgrade documentation.

I don't think there is any specific order but I'd probably start by the dedicated master nodes if you have that type of nodes.

3

@dadoonet I upgraded a coordination node first. My master nodes have x-pack license applied. However, I am still getting the following error.

[2019-09-08T23:51:04,568][INFO ][o.e.x.m.e.l.LocalExporter] [elkcoord02.mycluster] waiting for elected master node [{elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)

4

That's not an error message, just INFO, and in fact there's an open issue to remove them to avoid this confusion. These messages should stop once you've completed your upgrade.

1 Like
5

@DavidTurner I initially thought of ignoring the INFO message as well. However, I saw that the upgraded coordination node is unable to join the cluster. In monitoring app of Kibana, I find the other coordination nodes that are at previous version but not the upgraded one.

6

I am confused. The log message you quote indicates that it has successfully joined the cluster.

7

I suspect that the message indicates successful joining to the cluster. I restarted the node and the logs are as follows:

[2019-09-09T10:00:12,284][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-deprecation]
[2019-09-09T10:00:12,284][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-graph]
[2019-09-09T10:00:12,284][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-ilm]
[2019-09-09T10:00:12,284][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-logstash]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-ml]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-monitoring]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-rollup]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-security]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-sql]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-upgrade]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded module [x-pack-watcher]
[2019-09-09T10:00:12,285][INFO ][o.e.p.PluginsService     ] [elkcoord02.mycluster] loaded plugin [mapper-size]
[2019-09-09T10:00:16,843][INFO ][o.e.x.s.a.l.LdapUserSearchSessionFactory] [elkcoord02.mycluster] Realm [myrealm] is in user-search mode - base_dn=[my_dn], search filter=[(uid={0})]
[2019-09-09T10:00:17,164][INFO ][o.e.x.s.a.s.FileRolesStore] [elkcoord02.mycluster] parsed [10] roles from file [/etc/elasticsearch/roles.yml]
[2019-09-09T10:00:17,835][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [elkcoord02.mycluster] [controller/120607] [Main.cc@109] controller (64 bit): Version 6.8.2 (Build f4a9b28c0d5114) Copyright (c) 2019 Elasticsearch BV
[2019-09-09T10:00:18,485][DEBUG][o.e.a.ActionModule       ] [elkcoord02.mycluster] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-09-09T10:00:18,560][INFO ][o.e.d.DiscoveryModule    ] [elkcoord02.mycluster] using discovery type [zen] and host providers [settings]
[2019-09-09T10:00:19,482][INFO ][o.e.n.Node               ] [elkcoord02.mycluster] initialized
[2019-09-09T10:00:19,482][INFO ][o.e.n.Node               ] [elkcoord02.mycluster] starting ...
[2019-09-09T10:00:19,654][INFO ][o.e.t.TransportService   ] [elkcoord02.mycluster] publish_address {10.240.1.55:9401}, bound_addresses {[::]:9401}
[2019-09-09T10:00:19,667][INFO ][o.e.b.BootstrapChecks    ] [elkcoord02.mycluster] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-09-09T10:00:23,591][INFO ][o.e.c.s.ClusterApplierService] [elkcoord02.mycluster] detected_master {elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, added {{elkdatahot02.mycluster}{II5K2KjbQ_OXvJgBPo6sgw}{KQS6FVOuTQmP3DUSD0Blxw}[snipped] committed version [6727]])
[2019-09-09T10:00:24,695][INFO ][o.e.x.s.a.TokenService   ] [elkcoord02.mycluster] refresh keys
[2019-09-09T10:00:25,179][INFO ][o.e.x.s.a.TokenService   ] [elkcoord02.mycluster] refreshed keys
[2019-09-09T10:00:25,192][INFO ][o.e.l.LicenseService     ] [elkcoord02.mycluster] license [value] mode [snipped] - valid
[2019-09-09T10:00:25,222][INFO ][o.e.h.n.Netty4HttpServerTransport] [elkcoord02.mycluster] publish_address {10.240.1.55:9411}, bound_addresses {[::]:9411}
[2019-09-09T10:00:25,223][INFO ][o.e.n.Node               ] [elkcoord02.mycluster] started



[2019-09-09T10:00:50,131][INFO ][o.e.x.m.e.l.LocalExporter] [elkcoord02.mycluster] waiting for elected master node [{elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)

The node is still not part of the cluster. Am I missing something? Let me know if I can provide any other info.

8

@DavidTurner, @dadoonet As per the above logs, the master was detected but then why is the coordination node not added to the cluster?

Interestingly, we are only seeing an INFO log.

I am upgrading from 6.6.1 to 6.8.

9

What is making you think this? These logs indicate that this node joined the cluster. What does GET _cat/nodes report?

10

I can see from the monitoring dashboard that the node is not added.

53%20PM
Screen Shot 2019-09-10 at 12.12.53 PM.png944×534 45.7 KB

However, GET _cat/nodes reports that the node is present. That is my source of confusion. Please suggest.

11

I see. Hmm. I don't know why it's not picked up on the monitoring dashboard, but unfortunately that's not my area of expertise.

12

Can you suggest someone who could answer? Should I open an issue on Github?

13

Perhaps start with a new post in the Kibana forum since the question is no longer really about upgrade order.

14

It could be because you selected a specific timeframe in Kibana instead the last 15 minutes for example.

15

@dadoonet I have checked changing the time window from last 1 hour to last 4 hours and last 12 hours but the result is the same.

16

So may be it's a bug. As @DavidTurner wrote, you should ask in #kibana forum may be...

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.