# Tribe node can't connect after installing Shield **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-security\ **Created:** [July 3, 2015, 3:42pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877 "2015-07-03T15:42:17Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Mrc0113](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Mrc0113](https://discuss.elastic.co/u/Mrc0113)\ **Post date:** [July 3, 2015, 3:42pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/1 "2015-07-03T15:42:17Z")
\*Note that right now I only have 1 cluster up for my tribe nodes to connect to and before installing shield the tribe nodes could connect fine. After installing shield I created a system key and copied it to all nodes in the cluster then restarted each node in the cluster. The data/master nodes were able to join the cluster but the tribe nodes cannot. Multicast is disabled on all nodes and the node lists are correct in elasticsearch.yml. No shield specific configurations have been set in elasticsearch.yml so all defaults should be in place. I have not yet set up my keystores for node to node encryption. Only users and roles for HTTP access. I appreciate any help. The error I get in the tribe cluster log is as follows: [2015-07-03 11:25:55,997][WARN][discovery.zen.ping.unicast] [HOSTNAME/cluster-name] failed to send ping to [[#zen\_unicast\_2#][HOSTNAME][inet[HOSTNAME\_OF\_MASTER/IP\_OF\_MASTER:9300]]] org.elasticsearch.transport.RemoteTransportException: [HOSTNAME\_OF\_MASTER][inet[/\:9300]][internal:discovery/zen/unicast\_gte\_1\_4] Caused by: org.elasticsearch.shield.crypto.SignatureException: tampered signed text at org.elasticsearch.shield.crypto.InternalCryptoService.unsignAndVerify(InternalCryptoService.java:161) at org.elasticsearch.shield.authc.InternalAuthenticationService.authenticate(InternalAuthenticationService.java:99) at org.elasticsearch.shield.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:71) at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:171) at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:222) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:114) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.handler.ipfilter.IpFilteringHandlerImpl.handleUpstream(IpFilteringHandlerImpl.java:154) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 4, 2015, 3:56am UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/2 "2015-07-04T03:56:24Z")
It needs to be installed on _all_ nodes, including the Tribe one, did you do that? ---
**Author:** ![Mrc0113](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Mrc0113](https://discuss.elastic.co/u/Mrc0113)\ **Post date:** [July 4, 2015, 4:29am UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/3 "2015-07-04T04:29:39Z")
Yes - Shield was installed on all nodes and has the same system key on each. ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [July 7, 2015, 11:09am UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/4 "2015-07-07T11:09:05Z")
Can you try removing the system key from all nodes and then seeing if everything can connect? It appears as though the system key is not being recognized on some nodes while it may be recognized on others. ---
**Author:** ![Mrc0113](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Mrc0113](https://discuss.elastic.co/u/Mrc0113)\ **Post date:** [July 7, 2015, 1:56pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/5 "2015-07-07T13:56:22Z")
It seems like it is specifically an issue with Tribe nodes. This morning I changed the machines that couldn't connect from tribe nodes to just regular client nodes (http, no master, no data) and they were able to connect fine. Has anyone seen Tribe nodes work with the Shield version below? Shield info: ``` curl --user testuser:XXXXXX 'localhost:9200/_shield' { "status" : "enabled", "name" : "hostnamel", "cluster_name" : "cluster", "version" : { "number" : "1.2.1", "build_hash" : "f2cc2f1d3d7a0647412917d33a27890a1d958742", "build_timestamp" : "2015-04-29T16:46:24Z", "build_snapshot" : false }, "tagline" : "You know, for security" } ``` jaymode - I'll try as you requested as well. ---
**Author:** ![Mrc0113](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Mrc0113](https://discuss.elastic.co/u/Mrc0113)\ **Post date:** [July 7, 2015, 2:05pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/6 "2015-07-07T14:05:01Z")
jaymode, After removing the system key from all nodes the tribe nodes were able to join the cluster. ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [July 7, 2015, 5:40pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/7 "2015-07-07T17:40:03Z")
It seems somehow the system key is different on the nodes. Could you try to generate a new one and copy that one to all of the nodes? ---
**Author:** ![Mrc0113](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Mrc0113](https://discuss.elastic.co/u/Mrc0113)\ **Post date:** [July 7, 2015, 6:05pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/8 "2015-07-07T18:05:41Z")
@jaymode, The system key is the same on every node. I verified this using md5sum. Besides - if the system key was different then it shouldn't work when I change from using the tribe node to using a regular client node. ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [July 7, 2015, 7:00pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/9 "2015-07-07T19:00:31Z")
I'll try to reproduce this issue. What version of elasticsearch are you using? ---
**Author:** ![Mrc0113](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Mrc0113](https://discuss.elastic.co/u/Mrc0113)\ **Post date:** [July 7, 2015, 8:42pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/10 "2015-07-07T20:42:27Z")
Thanks! Elasticsearch: Version: 1.6.0, Build: cdd3ac4/2015-06-09T13:36:34Z, JVM: 1.8.0\_40 ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [July 8, 2015, 12:44pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/11 "2015-07-08T12:44:52Z")
Hi @Mrc0113, I spent some time today trying to reproduce the issue and was unable to do so; it appears to be working correctly for me. My config is the following: Elasticsearch version: ``` { "status" : 200, "name" : "Access", "cluster_name" : "tribe", "version" : { "number" : "1.6.0", "build_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0", "build_timestamp" : "2015-06-09T13:36:34Z", "build_snapshot" : false, "lucene_version" : "4.10.4" }, "tagline" : "You Know, for Search" } ``` Shield version: ``` { "status" : "enabled", "name" : "Access", "cluster_name" : "tribe", "version" : { "number" : "1.2.1", "build_hash" : "f2cc2f1d3d7a0647412917d33a27890a1d958742", "build_timestamp" : "2015-04-29T16:46:24Z", "build_snapshot" : false }, "tagline" : "You know, for security" } ``` I have two nodes. One is master only and the other is data only (also worked with both as master and data nodes). Master only config: ``` cluster.name: "c1" discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["localhost:9300", "localhost:9301"] transport.tcp.port: 9300 node.master: true node.data: false ``` Data only node config: ``` cluster.name: "c1" discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["localhost:9300", "localhost:9301"] transport.tcp.port: 9301 node.master: false ``` Then a single tribe node with the following config: ``` cluster.name: "tribe" discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: "localhost:9302" transport.tcp.port: 9302 tribe: c1: cluster.name: c1 discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: ["localhost:9300", "localhost:9301"] ``` Could you try a simple configuration like the above and see if it also works for you with the system key? Any more details on your installation, like if you installed from a RPM or Tar file? ---
**Author:** ![Mrc0113](https://avatars.discourse-cdn.com/v4/letter/m/ea666f/32.png) [@Mrc0113](https://discuss.elastic.co/u/Mrc0113)\ **Post date:** [July 8, 2015, 2:23pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/12 "2015-07-08T14:23:30Z")
@jaymode. Interesting. Thanks for trying. I installed via **rpm** on RHEL 6 w/ Java Oracle 1.8 I'll give it a shot with minimal config at some point over the next few days. { "status" : 200, "name" : "name", "cluster\_name" : "cluster", "version" : { "number" : "1.6.0", "build\_hash" : "cdd3ac4dde4f69524ec0a14de3828cb95bbb86d0", "build\_timestamp" : "2015-06-09T13:36:34Z", "build\_snapshot" : false, "lucene\_version" : "4.10.4" }, "tagline" : "You Know, for Search" } ---
**Author:** ![jaymode](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jaymode/32/50103_2.png) [@jaymode](https://discuss.elastic.co/u/jaymode)\ **Post date:** [July 8, 2015, 2:42pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/13 "2015-07-08T14:42:02Z")
The only other thing I can think of is that when the environment isn't configured correctly, the `system_key` would get placed in `/usr/share/elasticsearch/config/shield` and not in `/etc/elasticsearch/shield`. Maybe there is one node like that, but doesn't seem likely based on your previous comments about a node client working. ---
**Author:** ![Tony\_Glynn](https://avatars.discourse-cdn.com/v4/letter/t/c0e974/32.png) [@Tony\_Glynn](https://discuss.elastic.co/u/Tony_Glynn)\ **Post date:** [July 5, 2016, 12:56am UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/14 "2016-07-05T00:56:00Z")
Had the same problem. 2 nodes were /opt/elasticsearch 1 node was /opt/elastic. Hmm... need to my cfg act together. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:43pm UTC](https://discuss.elastic.co/t/tribe-node-cant-connect-after-installing-shield/24877/15 "2017-07-06T13:43:05Z")