I’m trying to build elasticsearch 6.5 cluster , mcluster consists of 3 nodes , I managed to installed the cluster without x-pack successfully and all 3 nodes appears in _cat API. However, when I configured x-pack only 2 nodes at a time can join the cluster successfully, for example [either 1& 2; 2 ,2& 3; 3 or 1&3] the first two will join normally the last node wont.
After couple of retries the last node will not be rejected from the cluster with below exceptions:
[2018-12-09T06:33:45,622][INFO ][o.e.c.s.ClusterApplierService] [master] added {{node-02}{HOKSU6z1RVaXx7WS_VWpmA}{4Chz9jZISZOz0_AluwBexA}{172.31.35.173}{172.31.35.173:9300}{ml.machine_memory=1926344704, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true, zone=2},}, reason: apply cluster state (from master [master {node-01}{EdOKMIeHQ_KSCc3yXTIk6Q}{JM3rOVNHR0u6cYf4_LOAmw}{172.31.16.192}{172.31.16.192:9300}{ml.machine_memory=1926344704, rack=2, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [473]])
=[2018-12-09T06:33:45,662][WARN ][o.e.c.NodeConnectionsService] [master] failed to connect to node {node-02}{HOKSU6z1RVaXx7WS_VWpmA}{4Chz9jZISZOz0_AluwBexA}{172.31.35.173}{172.31.35.173:9300}{ml.machine_memory=1926344704, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true, zone=2} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [node-02][172.31.35.173:9300] general node connection failure
at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:142) ~[elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:369) ~[elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:356) ~[elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.0.jar:6.5.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: java.lang.IllegalStateException: handshake failed with {node-02}{HOKSU6z1RVaXx7WS_VWpmA}{4Chz9jZISZOz0_AluwBexA}{172.31.35.173}{172.31.35.173:9300}{ml.machine_memory=1926344704, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true, zone=2}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:444) ~[elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.transport.TransportService.lambda$connectionValidator$4(TransportService.java:375) ~[elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:120) ~[elasticsearch-6.5.0.jar:6.5.0]
... 9 more
Caused by: org.elasticsearch.transport.RemoteTransportException: [node-02][172.31.35.173:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [internal:transport/handshake]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
at
org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:161) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:116) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:517) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$17(AuthenticationService.java:342) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:347) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:259) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$7(AuthenticationService.java:230) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:248) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:182) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.5.0.jar:6.5.0]
at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:310) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:209) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:220) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:174) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:134) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:104) ~[?:?]
at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:130) ~[?:?]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]