ES cluster with Shield - node not joining cluster

Elastic Stack Elasticsearch
1

I am trying to configure 2 nodes ELK cluster with Shield plugin enabled but cluster status says its yellow.
The version I am using is 2.3 for ELK. When I check the cluster status its NOT showing me 2 nodes where as I have configured unique cluster name. Also tried using unicast and multicast option but no luck.

The logs does not have any suspicious entries. I do see the entries on authenticated user and access granted entries. No errors or warnings.

If I put explicit node.master and node.data entries then on second node which is only data node I get following exception -

[2016-07-06 11:46:32,053][WARN ][discovery.zen.ping.unicast] [irldxvm022] failed to send ping to [{#zen_unicast_1#}{9.126.112.35}{9.126.112.35:9300}]
RemoteTransportException[[irldxvm002][9.126.112.35:9300][internal:discovery/zen/unicast]]; nested: IllegalArgumentException[tampered signed text];
Caused by: java.lang.IllegalArgumentException: tampered signed text

And on my first node which is explisitely configured master , I get following entries in the logs

[2016-07-06 11:48:45,773] [irldxvm002] [transport] [tampered_request] origin_type=[transport], origin_address=[9.126.112.72], action=[internal:discovery/zen/unicast]
[2016-07-06 11:48:47,274] [irldxvm002] [transport] [tampered_request] origin_type=[transport], origin_address=[9.126.112.72], action=[internal:discovery/zen/unicast]
[2016-07-06 11:48:47,278] [irldxvm002] [transport] [tampered_request] origin_type=[transport], origin_address=[9.126.112.72], action=[internal:discovery/zen/unicast]

Ping working from both machines as well as telnet to port 9300. Am I missing anything ?

2

Hi ,

I could resolve this issue by removing the system key from master server. We are not using tribe nodes.

Regards,
Vinod

3

You need the system key on all of the nodes. Did you add it to the new node? If not, that is why the node cannot join the cluster.

4

Yes that is correct, the key was not there on second machine. But we are not using tribe nodes, still I should add the system key on all nodes ?

5

Yes the system key has to be on ALL nodes as documented otherwise they cannot communicate with each other.