Cannot uninstall x-pack during rolling upgrade from Elasticsearch 5.6.10 to Elasticsearch 6.2.4

Elastic Stack Elasticsearch
1

I have an Elasticsearch 5.6.10 cluster (2 master, 2 client and 3 data nodes) with x-pack 5.6.10 installed. But all x-pack features are disabled with the following configurations in config\elasticsearch.yml files of all nodes.

node.ml: false
xpack.graph.enabled: false
xpack.ml.enabled: false
xpack.monitoring.enabled: false
xpack.security.enabled: false
xpack.watcher.enabled: false

Now, I want to upgrade to Elasticsearch 6.2.4 and remove x-pack feature completely. I want to do this through rolling upgrade (full cluster restart is not an option). I upgrade the first master node with the following steps:

  1. Stop and uninstall Elasticsearch 5.6.10.
  2. Copy Elasticsearch 6.2.4 folder into a new folder without the x-pack plugin. All the settings I mentioned above are removed in the config file of this version.
  3. Install and start Elasticsearch 6.2.4 from the new folder. It fails to join the cluster and my upgrade process is stuck here.

Below are the logs from this master node starting from just before the upgrade process to just after it fails to join the cluster:

[2018-11-07T11:47:53,022][WARN ][o.e.l.LicenseService     ] [es-m01-rm] 
#
# LICENSE [EXPIRED] ON [FRIDAY, AUGUST 10, 2018]. IF YOU HAVE A NEW LICENSE, PLEASE UPDATE IT.
# <removing for brevity>
[2018-11-07T11:51:32,404][INFO ][o.e.n.Node               ] [es-m01-rm] stopping ...
[2018-11-07T11:51:32,451][INFO ][o.e.n.Node               ] [es-m01-rm] stopped
[2018-11-07T11:51:32,451][INFO ][o.e.n.Node               ] [es-m01-rm] closing ...
[2018-11-07T11:51:32,467][INFO ][o.e.n.Node               ] [es-m01-rm] closed
[2018-11-07T11:59:31,141][INFO ][o.e.n.Node               ] [es-m01-rm] initializing ...
[2018-11-07T11:59:31,267][INFO ][o.e.e.NodeEnvironment    ] [es-m01-rm] using [1] data paths, mounts [[(F:)]], net usable_space [1021.6gb], net total_space [1021.8gb], types [NTFS]
[2018-11-07T11:59:31,267][INFO ][o.e.e.NodeEnvironment    ] [es-m01-rm] heap size [3.4gb], compressed ordinary object pointers [true]
[2018-11-07T11:59:31,376][INFO ][o.e.n.Node               ] [es-m01-rm] node name [es-m01-rm], node ID [CA_sk5BNQbG0uD-ogvEnAA]
[2018-11-07T11:59:31,376][INFO ][o.e.n.Node               ] [es-m01-rm] version[6.2.4], pid[4200], build[ccec39f/2018-04-12T20:37:28.497551Z], OS[Windows Server 2012 R2/6.3/amd64], JVM[Azul Systems, Inc./OpenJDK 64-Bit Server VM/1.8.0_72/25.72-b15]
[2018-11-07T11:59:31,376][INFO ][o.e.n.Node               ] [es-m01-rm] JVM arguments [<removing for brevity>]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [aggs-matrix-stats]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [analysis-common]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [ingest-common]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [lang-expression]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [lang-mustache]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [lang-painless]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [mapper-extras]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [parent-join]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [percolator]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [rank-eval]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [reindex]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [repository-url]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [transport-netty4]
[2018-11-07T11:59:32,329][INFO ][o.e.p.PluginsService     ] [es-m01-rm] loaded module [tribe]
[2018-11-07T11:59:37,063][WARN ][o.e.c.m.MetaData         ] Skipping unknown custom object with type licenses
[2018-11-07T11:59:37,079][WARN ][o.e.c.m.MetaData         ] Skipping unknown custom object with type ml
[2018-11-07T11:59:39,048][WARN ][o.e.c.m.MetaData         ] Skipping unknown custom object with type licenses
[2018-11-07T11:59:39,048][WARN ][o.e.c.m.MetaData         ] Skipping unknown custom object with type ml
[2018-11-07T11:59:39,360][INFO ][o.e.d.DiscoveryModule    ] [es-m01-rm] using discovery type [zen]
[2018-11-07T11:59:39,954][INFO ][o.e.n.Node               ] [es-m01-rm] initialized
[2018-11-07T11:59:39,954][INFO ][o.e.n.Node               ] [es-m01-rm] starting ...
[2018-11-07T11:59:40,267][INFO ][o.e.t.TransportService   ] [es-m01-rm] publish_address {192.168.0.111:9300}, bound_addresses {[::]:9300}
[2018-11-07T11:59:40,282][WARN ][o.e.c.m.MetaData         ] Skipping unknown custom object with type licenses
[2018-11-07T11:59:40,298][WARN ][o.e.c.m.MetaData         ] Skipping unknown custom object with type ml
[2018-11-07T11:59:40,454][INFO ][o.e.b.BootstrapChecks    ] [es-m01-rm] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-11-07T12:00:05,116][INFO ][o.e.d.z.ZenDiscovery     ] [es-m01-rm] failed to send join request to master [{es-m02-rm}{0rY04gblSYqkfvK0ObSjiQ}{gL9J1sALTJOPZm2pnGdDYQ}{192.168.0.102}{192.168.0.102:9300}{faultDomain=1, updateDomain=1}], reason [RemoteTransportException[[es-m02-rm][192.168.0.102:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[es-m01-rm][192.168.0.111:9300] connect_timeout[30s]]; nested: IOException[Connection timed out: no further information: 192.168.0.111/192.168.0.111:9300]; nested: IOException[Connection timed out: no further information]; ]
[2018-11-07T12:00:10,523][WARN ][o.e.n.Node               ] [es-m01-rm] timed out while waiting for initial discovery state - timeout: 30s
[2018-11-07T12:00:10,523][INFO ][o.e.h.n.Netty4HttpServerTransport] [es-m01-rm] publish_address {192.168.0.111:9200}, bound_addresses {[::]:9200}
[2018-11-07T12:00:10,523][INFO ][o.e.n.Node               ] [es-m01-rm] started
[2018-11-07T12:00:29,148][INFO ][o.e.d.z.ZenDiscovery     ] [es-m01-rm] failed to send join request to master [{es-m02-rm}{0rY04gblSYqkfvK0ObSjiQ}{gL9J1sALTJOPZm2pnGdDYQ}{192.168.0.102}{192.168.0.102:9300}{faultDomain=1, updateDomain=1}], reason [RemoteTransportException[[es-m02-rm][192.168.0.102:9300][internal:discovery/zen/join]]; nested: ConnectTransportException[[es-m01-rm][192.168.0.111:9300] connect_timeout[30s]]; nested: IOException[Connection timed out: no further information: 192.168.0.111/192.168.0.111:9300]; nested: IOException[Connection timed out: no further information]; ]

How do I get past this problem?

2

This is going to be tricky to do - once you have X-Pack objects in your cluster state, it's very difficult to remove x-pack.

Why exactly do you want to do this?
From Elasticsearch 6.3 onwards, X-Pack is automatically included in the default distribution, so you're swimming against the tide here.

3

Understood. We don't use any x-pack features anymore and hence wanted to clean up the cluster. Now I guess we'll have to keep the plugin installed with disabled features forever till we do a full cluster restart.

4

Have you seen the most recent X-Pack features such as Rollups & SQL? (and there are more coming).
You are of course free to use a non-X-Pack version of Elasticsearch, but you will be cutting yourself off
from a range of new (free) features that might be of help to you.

A full cluster restart won't solve this for you. The cluster state is persisted across restarts, so it will be a challenge to remove all traces of X-Pack even if you do a full restart.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.