Does rolling upgrade node sequence matter?


(None) #1

Hi, I'm trying to upgrade from 6.1.2 to 6.2.2

I have separate nodes: masters, ingest, coordinators and data.

I started by upgrading the master nodes.

But I get the following message in the logs... I do have x-pack installed, but basic license.

[2018-03-05T00:51:25,599][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2018-03-05T00:51:26,333][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2018-03-05T00:51:26,951][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2018-03-05T00:51:27,628][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2018-03-05T00:51:28,643][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2018-03-05T00:51:29,768][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)
[2018-03-05T00:57:29,672][INFO ][o.e.c.s.ClusterApplierService] [master] added {{master}{xOkFTYh9Rqmh-0T73adWMg}{5_cb0oA4TViDQvPhEYpUcg}{xxxxxx.134}{xxxxxx.134:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, ml.enabled=true},}, reason: apply cluster state (from master [master {master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true} committed version [26688]])
[2018-03-05T00:57:33,843][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)

Any action keeps printing this on the master node. Example removing one of the master nodes...

[2018-03-05T01:06:33,900][INFO ][o.e.c.s.ClusterApplierService] [master] removed {{master}{xOkFTYh9Rqmh-0T73adWMg}{5_cb0oA4TViDQvPhEYpUcg}{xxxxxx.134}{xxxxxx.134:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, ml.enabled=true},}, reason: apply cluster state (from master [master {master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{9.0.12.130}{9.0.12.130:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true} committed version [26689]])
[2018-03-05T01:06:33,905][WARN ][o.e.c.NodeConnectionsService] [master] failed to connect to node {master}{xOkFTYh9Rqmh-0T73adWMg}{5_cb0oA4TViDQvPhEYpUcg}{xxxxxx.134}{xxxxxx.134:9300}{ml.machine_memory=4294967296, ml.max_open_jobs=20, ml.enabled=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [master][xxxxxx.134:9300] connect_exception
	at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:616) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:513) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:331) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:318) ~[elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:154) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.cluster.NodeConnectionsService$ConnectionChecker.doRun(NodeConnectionsService.java:183) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.2.jar:6.2.2]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.2.jar:6.2.2]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_161]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_161]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: xxxxxx.134/xxxxxx.134:9300
	at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:323) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:340) ~
...
[2018-03-05T01:06:34,066][INFO ][o.e.x.m.e.l.LocalExporter] waiting for elected master node [{master}{AwR3s0L-QIOVSCPYWMuhIw}{KvG8KoaiStmxqWvX0pcOuQ}{xxxxxx.130}{xxxxxx:9300}{ml.machine_memory=3221225472, ml.max_open_jobs=20, ml.enabled=true}] to setup local exporter [default_local] (does it have x-pack installed?)

(None) #2

Hi, has anyone experienced this before?


(None) #3

@warkolm Hi, Mark I know you are a busy guy. Last time I didn't have the deployment logs, but now I do. Let me know if you have a bit of time to check this. Thanks


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.