Node not able to join the cluster after enabling x-pack in elastic v6.5

Elastic Stack Elasticsearch
1

I’m trying to build elasticsearch 6.5 cluster , mcluster consists of 3 nodes , I managed to installed the cluster without x-pack successfully and all 3 nodes appears in _cat API. However, when I configured x-pack only 2 nodes at a time can join the cluster successfully, for example [either 1& 2; 2 ,2& 3; 3 or 1&3] the first two will join normally the last node wont.

After couple of retries the last node will not be rejected from the cluster with below exceptions:

       [2018-12-09T06:33:45,622][INFO ][o.e.c.s.ClusterApplierService] [master] added {{node-02}{HOKSU6z1RVaXx7WS_VWpmA}{4Chz9jZISZOz0_AluwBexA}{172.31.35.173}{172.31.35.173:9300}{ml.machine_memory=1926344704, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true, zone=2},}, reason: apply cluster state (from master [master {node-01}{EdOKMIeHQ_KSCc3yXTIk6Q}{JM3rOVNHR0u6cYf4_LOAmw}{172.31.16.192}{172.31.16.192:9300}{ml.machine_memory=1926344704, rack=2, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} committed version [473]])
    =[2018-12-09T06:33:45,662][WARN ][o.e.c.NodeConnectionsService] [master] failed to connect to node {node-02}{HOKSU6z1RVaXx7WS_VWpmA}{4Chz9jZISZOz0_AluwBexA}{172.31.35.173}{172.31.35.173:9300}{ml.machine_memory=1926344704, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true, zone=2} (tried [1] times)
    org.elasticsearch.transport.ConnectTransportException: [node-02][172.31.35.173:9300] general node connection failure
            at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:142) ~[elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:369) ~[elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:356) ~[elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.0.jar:6.5.0]
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
            at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
    Caused by: java.lang.IllegalStateException: handshake failed with {node-02}{HOKSU6z1RVaXx7WS_VWpmA}{4Chz9jZISZOz0_AluwBexA}{172.31.35.173}{172.31.35.173:9300}{ml.machine_memory=1926344704, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true, zone=2}
            at org.elasticsearch.transport.TransportService.handshake(TransportService.java:444) ~[elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.transport.TransportService.lambda$connectionValidator$4(TransportService.java:375) ~[elasticsearch-6.5.0.jar:6.5.0]
            at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:120) ~[elasticsearch-6.5.0.jar:6.5.0]
            ... 9 more
    Caused by: org.elasticsearch.transport.RemoteTransportException: [node-02][172.31.35.173:9300][internal:transport/handshake]
    Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [internal:transport/handshake]
            at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
            at
org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:161) ~[?:?]
        at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:116) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:517) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$17(AuthenticationService.java:342) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:347) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:259) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$7(AuthenticationService.java:230) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken(AuthenticationService.java:248) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:182) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-6.5.0.jar:6.5.0]
        at org.elasticsearch.xpack.security.authc.TokenService.getAndValidateToken(TokenService.java:310) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:178) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$4(AuthenticationService.java:209) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:220) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:174) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:134) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:104) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:130) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:307) ~[?:?]
2

Please don't post unformatted logs as they're very hard to read.

Instead paste the text and format it with </> icon, and check the preview
window to make sure it's properly formatted before posting it. This makes it
more likely that your question will receive a useful answer.

It would be great if you could update your post to solve this.

3

Assuming you want to apply security to your cluster, then you should add

xpack.security.enabled: true
xpack.license.self_generated.type: trial

to your elasticsearch.yml

It looks like you're running into an issue where you have a mismatch where some nodes think security is enabled and some don't. The settings above should force all nodes to have the same configuration.