Hi!
We have been using es 1.7.2 for a while, and have just stood up two new es 5.1.2 clusters (one main, and one monitoring). Had everything working without xpack. Then installed xpack into elasticsearch) on all the main cluster nodes and configured the exporter settings to send data to the monitoring cluster. Installed xpack into the es and kibana on the monitoring clusters, and that cluster is working fine and is getting data from the main cluster. However, the client nodes in the main cluster now won't start elasticsearch. I currently only have xpack installed into elasticsearch on the client nodes on the main cluster (since it's not the monitoring cluster), and if I do not set xpack.security.enabled: false in the elasticsearch.yml, when I try to access ip:9200 to confirm es is running and listening, I get prompted for a login. I don't want to use a login (and I've disabled security on the monitoring cluster). But, when I DO set xpack.security.enabled: false in the elasticsearch.yml, es won't start and I get an error from the master in the log: "][internal:transport/handshake] Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [internal:transport/handshake]". Including the more of the error below, but not the full error because this capped me at 6000 characters.
I don't know why this is happening on the client nodes. It's not happening on the master or data nodes, and the only xpack setting configured there is xpack.monitoring.exporters.
I'll likely create a support ticket, but I thought I'd reach out this way because I might get a quicker answer.
[2017-01-25T21:18:40,867][WARN ][o.e.d.z.UnicastZenPing ] [clientnodename] [1] failed send ping to {#zen_unicast_1#}{_2vPSOnCRdaEk1xMVy0qbw}{ip}{ip:9300}
java.lang.IllegalStateException: handshake failed with {#zen_unicast_1#}{_2vPSOnCRdaEk1xMVy0qbw}{ip}{ip:9300}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:345) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.transport.TransportService.connectToNodeLightAndHandshake(TransportService.java:319) ~[elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.discovery.zen.UnicastZenPing$2.run(UnicastZenPing.java:473) [elasticsearch-5.1.2.jar:5.1.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.2.jar:5.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.transport.RemoteTransportException: [masternodename][ip:9300][internal:transport/handshake]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [internal:transport/handshake]
at org.elasticsearch.xpack.security.support.Exceptions.authenticationError(Exceptions.java:39) ~[?:?]
at org.elasticsearch.xpack.security.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:74) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableTransportRequest.anonymousAccessDenied(AuthenticationService.java:506) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$handleNullToken$13(AuthenticationService.java:300) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.handleNullToken(AuthenticationService.java:305) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeToken(AuthenticationService.java:245) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$extractToken$5(AuthenticationService.java:221) ~[?:?]
at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.extractToken