Error when adding node into cluster - Failed to send join request to master

Elastic Stack Elasticsearch
1

Hi,
I am creating multiple nodes under one cluster. before to set up ES on production Server, i m doing it on my local machine-
Got tar file of ES and unzipped it and put same in two directorories(D1,D2)
and also install ES as service on same machine
Edit config file in all location(D1,D2 and /etc/elasticsearch/elasticsearch.yml) with -
in D1 - set cluster name as test-cluster and node name as node-1
in D2 - set cluster name as test-cluster and node name as node-2
in /etc/elasticsearch/elasticsearch.yml - set cluster name as test-cluster and node name as node-3
started ES instance in D2 and it is successfully started as master node
now started ES instance in D1 and it is successfully started and joined the running cluster
now started Es instance as a service and it is successfully started and but not able join cluster and getting following error -
error in logs on node-3-
[2016-07-13 19:03:23,093][WARN ][transport.netty ] [node-3] exception caught on transport layer [[id: 0xc73d8b09, /127.0.0.1:39331 => /127.0.0.1:9302]], closing connection
java.lang.IllegalStateException: Message not fully read (request) for requestId [11508], action [internal:discovery/zen/join/validate], readerIndex [7874] vs expected [9381]; resetting
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:121)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-07-13 19:03:23,096][INFO ][discovery.zen ] [node-3] failed to send join request to master [{node-2}{nZ1u0minT06lDpOIVC0rhQ}{127.0.0.1}{127.0.0.1:9301}], reason [RemoteTransportException[[node-2][127.0.0.1:9301][internal:discovery/zen/join]]; nested: IllegalStateException[failure when sending a validation request to node]; nested: RemoteTransportException[[node-3][127.0.0.1:9302][internal:discovery/zen/join/validate]]; nested: IllegalArgumentException[No custom metadata prototype registered for type [licenses], node like missing plugins]; ]

and in node-1 logs error is -
[2016-07-13 19:24:52,579][WARN ][discovery.zen ] [node-2] failed to validate incoming join request from node [{node-3}{6LgUEyWZQmeYuFoCSZ5EUA}{127.0.0.1}{127.0.0.1:9302}]

Any kind help will be much appreciated.

Thanks,
Satty

2

I am using ES 2.3.2

3

Apparently you did not install license plugin on all nodes? No custom metadata prototype registered for type [licenses], node like missing plugins

4

Oh and BTW why do you install 2 nodes on the same machine?

5

Hi Dadoonet,
I am doing it on my local machine and to test this scenario .. i need to test it on my local machine.

So i m running 3 ES instances on my machine and all are running fine but one instance which is started as a service which config file located in "/etc/elasticsearch" is not joining cluster.

I went through ES doc to add nodes(running on different machines) into cluster- https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#common-network-settings

It is saying no need to change network config settings if all nodes are running on same machine.

6

So don't run instances as services. Just run them from the elasticsearch directory.

Basically launch in 2 windows:

bin/elasticsearch

And that's all.