Hi ,
Recently I upgraded elastic, logstash and kibana to 6.4.0 from 6.2.x. After restarting es nodes and tribe, getting below exception. Due to which sync is not happening resulting to stopping indexing. I disabled xpack.security.enabled : false in all nodes. Before installing 6.4.0 i deleted old elasticsearch.keystore and new one got generated automatically while starting up the server. Please someone help here what security/authentication configurations are required.
Exception stack trace
[2018-09-14T10:42:34,581][WARN ][o.e.c.NodeConnectionsService] [prod_90_01] failed to connect to node {prod_tribe_02_01/t1}{uWwnGOoPTxGVnvXYLMxOVA}{gi6CELwVRyS6z2GsYk66pQ}{xxxx90.corp.xxxx.com}{10.192.169.92:9300}{xpack.installed=true} (tried [1] times)
org.elasticsearch.transport.ConnectTransportException: [prod_tribe___02_01/t1][10.192.169.92:9300] general node connection failure
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:571) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:329) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:316) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.NodeConnectionsService.validateAndConnectIfNeeded(NodeConnectionsService.java:153) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.cluster.NodeConnectionsService$1.doRun(NodeConnectionsService.java:106) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) [elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.4.0.jar:6.4.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.IllegalStateException: handshake failed with {prod_tribe___02_01/t1}{uWwnGOoPTxGVnvXYLMxOVA}{gi6CELwVRyS6z2GsYk66pQ}{xxxx90.corp.xxxx.com}{10.192.169.92:9300}{xpack.installed=true}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:400) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.transport.TransportService.lambda$connectToNode$4(TransportService.java:331) ~[elasticsearch-6.4.0.jar:6.4.0]
at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:543) ~[elasticsearch-6.4.0.jar:6.4.0]
... 9 more
Caused by: org.elasticsearch.transport.RemoteTransportException: [prod_tribe___02_01/t1][10.192.169.92:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [internal:transport/handshake]
at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:18) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.createAuthenticationError(DefaultAuthenticationFailureHandler.java:129) ~[?:?]
at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:84) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:526) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$17(AuthenticationService.java:337) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:342) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:255) ~[?:?]