# Upgrade order

**URL:** <https://discuss.elastic.co/t/upgrade-order/198627>\
**Category:** Elasticsearch\
**Created:** [September 9, 2019, 8:07am UTC](https://discuss.elastic.co/t/upgrade-order/198627 "2019-09-09T08:07:30Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [September 9, 2019, 8:07am UTC](https://discuss.elastic.co/t/upgrade-order/198627/1 "2019-09-09T08:07:30Z")

</div>

When upgrading from any version to a higher version that is supported by Elastic, what is the order of upgrade for node types: master, index-only, coordination-only ?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 9, 2019, 8:36am UTC](https://discuss.elastic.co/t/upgrade-order/198627/2 "2019-09-09T08:36:13Z")

</div>

I'd read the [upgrade guide](https://www.elastic.co/guide/en/elastic-stack/current/upgrading-elastic-stack.html).

This links also to the specific [elasticsearch upgrade documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/setup-upgrade.html).

I don't think there is any specific order but I'd probably start by the dedicated master nodes if you have that type of nodes.

---

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [September 9, 2019, 10:41am UTC](https://discuss.elastic.co/t/upgrade-order/198627/3 "2019-09-09T10:41:29Z")

</div>

@dadoonet I upgraded a coordination node first. My master nodes have x-pack license applied. However, I am still getting the following error.

`[2019-09-08T23:51:04,568][INFO][o.e.x.m.e.l.LocalExporter] [elkcoord02.mycluster] waiting for elected master node [{elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)`

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 9, 2019, 12:56pm UTC](https://discuss.elastic.co/t/upgrade-order/198627/4 "2019-09-09T12:56:36Z")

</div>

That's not an error message, just `INFO`, and in fact there's an [open issue to remove them](https://github.com/elastic/elasticsearch/issues/40898) to avoid this confusion. These messages should stop once you've completed your upgrade.

---

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [September 9, 2019, 1:41pm UTC](https://discuss.elastic.co/t/upgrade-order/198627/5 "2019-09-09T13:41:46Z")

</div>

@DavidTurner I initially thought of ignoring the `INFO` message as well. However, I saw that the upgraded coordination node is unable to join the cluster. In monitoring app of Kibana, I find the other coordination nodes that are at previous version but not the upgraded one.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 9, 2019, 3:38pm UTC](https://discuss.elastic.co/t/upgrade-order/198627/6 "2019-09-09T15:38:39Z")

</div>

> [@amitavmohanty01](#):
>
> the upgraded coordination node is unable to join the cluster

I am confused. The log message you quote indicates that it has successfully joined the cluster.

---

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [September 9, 2019, 6:34pm UTC](https://discuss.elastic.co/t/upgrade-order/198627/7 "2019-09-09T18:34:27Z")

</div>

> [@DavidTurner](#):
>
> I am confused. The log message you quote indicates that it has successfully joined the cluster.

I suspect that the message indicates successful joining to the cluster. I restarted the node and the logs are as follows:

```
[2019-09-09T10:00:12,284][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-deprecation]
[2019-09-09T10:00:12,284][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-graph]
[2019-09-09T10:00:12,284][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-ilm]
[2019-09-09T10:00:12,284][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-logstash]
[2019-09-09T10:00:12,285][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-ml]
[2019-09-09T10:00:12,285][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-monitoring]
[2019-09-09T10:00:12,285][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-rollup]
[2019-09-09T10:00:12,285][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-security]
[2019-09-09T10:00:12,285][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-sql]
[2019-09-09T10:00:12,285][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-upgrade]
[2019-09-09T10:00:12,285][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded module [x-pack-watcher]
[2019-09-09T10:00:12,285][INFO][o.e.p.PluginsService] [elkcoord02.mycluster] loaded plugin [mapper-size]
[2019-09-09T10:00:16,843][INFO][o.e.x.s.a.l.LdapUserSearchSessionFactory] [elkcoord02.mycluster] Realm [myrealm] is in user-search mode - base_dn=[my_dn], search filter=[(uid={0})]
[2019-09-09T10:00:17,164][INFO][o.e.x.s.a.s.FileRolesStore] [elkcoord02.mycluster] parsed [10] roles from file [/etc/elasticsearch/roles.yml]
[2019-09-09T10:00:17,835][INFO][o.e.x.m.p.l.CppLogMessageHandler] [elkcoord02.mycluster] [controller/120607] [Main.cc@109] controller (64 bit): Version 6.8.2 (Build f4a9b28c0d5114) Copyright (c) 2019 Elasticsearch BV
[2019-09-09T10:00:18,485][DEBUG][o.e.a.ActionModule] [elkcoord02.mycluster] Using REST wrapper from plugin org.elasticsearch.xpack.security.Security
[2019-09-09T10:00:18,560][INFO][o.e.d.DiscoveryModule] [elkcoord02.mycluster] using discovery type [zen] and host providers [settings]
[2019-09-09T10:00:19,482][INFO][o.e.n.Node] [elkcoord02.mycluster] initialized
[2019-09-09T10:00:19,482][INFO][o.e.n.Node] [elkcoord02.mycluster] starting ...
[2019-09-09T10:00:19,654][INFO][o.e.t.TransportService] [elkcoord02.mycluster] publish_address {10.240.1.55:9401}, bound_addresses {[::]:9401}
[2019-09-09T10:00:19,667][INFO][o.e.b.BootstrapChecks] [elkcoord02.mycluster] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-09-09T10:00:23,591][INFO][o.e.c.s.ClusterApplierService] [elkcoord02.mycluster] detected_master {elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, added {{elkdatahot02.mycluster}{II5K2KjbQ_OXvJgBPo6sgw}{KQS6FVOuTQmP3DUSD0Blxw}[snipped] committed version [6727]])
[2019-09-09T10:00:24,695][INFO][o.e.x.s.a.TokenService] [elkcoord02.mycluster] refresh keys
[2019-09-09T10:00:25,179][INFO][o.e.x.s.a.TokenService] [elkcoord02.mycluster] refreshed keys
[2019-09-09T10:00:25,192][INFO][o.e.l.LicenseService] [elkcoord02.mycluster] license [value] mode [snipped] - valid
[2019-09-09T10:00:25,222][INFO][o.e.h.n.Netty4HttpServerTransport] [elkcoord02.mycluster] publish_address {10.240.1.55:9411}, bound_addresses {[::]:9411}
[2019-09-09T10:00:25,223][INFO][o.e.n.Node] [elkcoord02.mycluster] started

[2019-09-09T10:00:50,131][INFO][o.e.x.m.e.l.LocalExporter] [elkcoord02.mycluster] waiting for elected master node [{elkmaster03.mycluster}{kpmhJBoYSQ6ocWgEPo-kTA}{_dNLkO_nTDKMrDxPdILnug}{elkmaster03.mycluster.mydomain.com}{10.241.4.119:9401}{ml.machine_memory=8182050816, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)

```

The node is still not part of the cluster. Am I missing something? Let me know if I can provide any other info.

---

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [September 9, 2019, 8:39pm UTC](https://discuss.elastic.co/t/upgrade-order/198627/8 "2019-09-09T20:39:40Z")

</div>

@DavidTurner, @dadoonet As per the above logs, the master was detected but then why is the coordination node not added to the cluster?

Interestingly, we are only seeing an `INFO` log.

I am upgrading from 6.6.1 to 6.8.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 9, 2019, 9:58pm UTC](https://discuss.elastic.co/t/upgrade-order/198627/9 "2019-09-09T21:58:52Z")

</div>

> [@amitavmohanty01](#):
>
> The node is still not part of the cluster.

What is making you think this? These logs indicate that this node joined the cluster. What does `GET _cat/nodes` report?

---

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [September 10, 2019, 7:10am UTC](https://discuss.elastic.co/t/upgrade-order/198627/10 "2019-09-10T07:10:20Z")

</div>

I can see from the monitoring dashboard that the node is not added.

 ![53%20PM](https://us1.discourse-cdn.com/elastic/original/3X/4/f/4f9cd79642f42d3f5827081d9032c8ba7d37597b.png)

However, `GET _cat/nodes` reports that the node is present. That is my source of confusion. Please suggest.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 10, 2019, 7:38am UTC](https://discuss.elastic.co/t/upgrade-order/198627/11 "2019-09-10T07:38:40Z")

</div>

I see. Hmm. I don't know why it's not picked up on the monitoring dashboard, but unfortunately that's not my area of expertise.

---

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [September 10, 2019, 8:52am UTC](https://discuss.elastic.co/t/upgrade-order/198627/12 "2019-09-10T08:52:30Z")

</div>

Can you suggest someone who could answer? Should I open an issue on Github?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [September 10, 2019, 10:28am UTC](https://discuss.elastic.co/t/upgrade-order/198627/13 "2019-09-10T10:28:25Z")

</div>

Perhaps start with a new post in the [Kibana forum](https://discuss.elastic.co/c/kibana) since the question is no longer really about upgrade order.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 10, 2019, 11:25am UTC](https://discuss.elastic.co/t/upgrade-order/198627/14 "2019-09-10T11:25:04Z")

</div>

It could be because you selected a specific timeframe in Kibana instead the last 15 minutes for example.

---

<div class="post-metadata">

**Author:** ![amitavmohanty01](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amitavmohanty01/32/58017_2.png) [@amitavmohanty01](https://discuss.elastic.co/u/amitavmohanty01)\
**Post date:** [September 10, 2019, 7:38pm UTC](https://discuss.elastic.co/t/upgrade-order/198627/15 "2019-09-10T19:38:47Z")

</div>

@dadoonet I have checked changing the time window from last 1 hour to last 4 hours and last 12 hours but the result is the same.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [September 11, 2019, 6:40am UTC](https://discuss.elastic.co/t/upgrade-order/198627/16 "2019-09-11T06:40:42Z")

</div>

So may be it's a bug. As @DavidTurner wrote, you should ask in #kibana forum may be...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 9, 2019, 6:43am UTC](https://discuss.elastic.co/t/upgrade-order/198627/17 "2019-10-09T06:43:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.