ES cluster with Shield - node not joining cluster

I am trying to configure 2 nodes ELK cluster with Shield plugin enabled but cluster status says its yellow.
The version I am using is 2.3 for ELK. When I check the cluster status its NOT showing me 2 nodes where as I have configured unique cluster name. Also tried using unicast and multicast option but no luck.

The logs does not have any suspicious entries. I do see the entries on authenticated user and access granted entries. No errors or warnings.

If I put explicit node.master and entries then on second node which is only data node I get following exception -

[2016-07-06 11:46:32,053][WARN ][] [irldxvm022] failed to send ping to [{#zen_unicast_1#}{}{}]
RemoteTransportException[[irldxvm002][][internal:discovery/zen/unicast]]; nested: IllegalArgumentException[tampered signed text];
Caused by: java.lang.IllegalArgumentException: tampered signed text

And on my first node which is explisitely configured master , I get following entries in the logs

[2016-07-06 11:48:45,773] [irldxvm002] [transport] [tampered_request] origin_type=[transport], origin_address=[], action=[internal:discovery/zen/unicast]
[2016-07-06 11:48:47,274] [irldxvm002] [transport] [tampered_request] origin_type=[transport], origin_address=[], action=[internal:discovery/zen/unicast]
[2016-07-06 11:48:47,278] [irldxvm002] [transport] [tampered_request] origin_type=[transport], origin_address=[], action=[internal:discovery/zen/unicast]

Ping working from both machines as well as telnet to port 9300. Am I missing anything ?

Hi ,

I could resolve this issue by removing the system key from master server. We are not using tribe nodes.


You need the system key on all of the nodes. Did you add it to the new node? If not, that is why the node cannot join the cluster.

Yes that is correct, the key was not there on second machine. But we are not using tribe nodes, still I should add the system key on all nodes ?

Yes the system key has to be on ALL nodes as documented otherwise they cannot communicate with each other.