Elasticsearch -2.3.0 BindTransportException on EC2

Elastic Stack Elasticsearch
1

Hello,

This is first time I am using elasticsearch . I am using AWS as my infrastructure.

I have launched 4 EC2 Instances with the following purpose:

  1. Instance 1 : Elasticsearch Client
  2. Instance 2 : Elasticsearch Master
  3. Instance 3 : Elasticsearch Data Node - 1
  4. Instance 4 : Elasticsearch Data Node - 2

The following elastic-ips are used in the below section just for better understanding of configuration files

Elastic - ip

1.2.3.4 = Instance - 1

5.6.7.8 = Instance - 2

9.10.11.12 = Instance - 3

13.14.15.16 = Instance - 4

I am not using aws plugin for the configurations.

I have configured the following parameters of elasticsearch.yml file on each of the instance as follows:

Instance 1 : Elasticsearch Client

  1. cluster.name my-clustername
  2. node.name: Client
  3. node.master: false
  4. node.data: false
  5. path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0
  6. network.host: 1.2.3.4 [Elastic-ip of my instance]
  7. discovery.zen.ping.multicast.enabled: false
  8. discovery.zen.ping.unicast.hosts: ["1.2.3.4" , "5.6.7.8" , "9.10.11.12", "13.14.15.16"]

Instance 2 : Elasticsearch Master

  1. cluster.name my-clustername
  2. node.name: Master
  3. node.master: true
  4. node.data: false
  5. path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0
  6. network.host: 5.6.7.8 [Elastic-ip of my instance]
  7. discovery.zen.ping.multicast.enabled: false
  8. discovery.zen.ping.unicast.hosts: ["1.2.3.4" , "5.6.7.8" , "9.10.11.12", "13.14.15.16"]

Instance 3 : Elasticsearch Data Node -1

  1. cluster.name my-clustername
  2. node.name: Slave_1
  3. node.master: false
  4. node.data: true
  5. path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0
  6. network.host: 9.10.11.12 [Elastic-ip of my instance]
  7. discovery.zen.ping.multicast.enabled: false
  8. discovery.zen.ping.unicast.hosts: ["1.2.3.4" , "5.6.7.8" , "9.10.11.12", "13.14.15.16"]

Instance 4 : Elasticsearch Data Node -2

  1. cluster.name my-clustername
  2. node.name: Slave_2
  3. node.master: false
  4. node.data: true
  5. path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0
  6. network.host: 13.14.15.16 [Elastic-ip of my instance]
  7. discovery.zen.ping.multicast.enabled: false
  8. discovery.zen.ping.unicast.hosts: ["1.2.3.4" , "5.6.7.8" , "9.10.11.12", "13.14.15.16"]

After all these setting when I start my elasicsearch service on my client instance I am facing the following exception.

Exception in thread "main" BindTransportException[Failed to bind to [9300-9400]]; nested: ChannelException[Failed to bind to: /1.2.3.4:9400]; nested: BindException[Cannot assign requested address];
Likely root cause: java.net.BindException: Cannot assign requested address
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:391)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:315)
at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Refer to the log for complete error details.

The above error I am getting on each of my instance when I start elasticsearch service.
I am not understanding the root cause and reason for this error and what is the solution to overcome this problem.

I hope I am clear with my question and other details.

Thank you.

2

Why not?

There's no such thing as a slave in ES, you may want to change it.

Also, in general, having a single node with multiple data nodes is not a good idea, see Important Configuration Changes | Elasticsearch: The Definitive Guide [2.x] | Elastic

This is pointless in 2.X, what version are you on?

Where is it getting 9400 from? Have you set an interface for ES to listen on?

3

Thank you for your response.

  1. I am using version 2.3.0.
  2. I did not mentioned port number anywhere is my elasticsearch.yml file you can see clearly in my above configuration.
  3. I am not using plugin because I want to understand the properties and its use.

Thank you

4

Is that the entire config?

5

Yes Sir, the properties which I have shown are the only properties which I have changed.
Thank you

6

Then you need to set network.host as well - https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#common-network-settings

7

Sir please look at the point 6 in each of the configuration which I listed above.
I have configured network.host to the elastic ip of each of my instance respectively.
I hope I am doing nothing wrong by setting network.host to the elastic-ip of the instance.
And I also I have changed node.name to DataNode_1 :slight_smile:
Thank you

8

Oh right, sorry your config is really hard to read like that.

9

Hello,

I am using EC2 plugin now and I have made the following configuration in my elasticsearch.yml.

cluster.name: mycluster

node.name: Master_1
node.master: true
node.data: false

path.data: /home/ubuntu/elasticsearch/data/elasticsearch/nodes/0

network.host: _ec2_

cloud:
aws:
access_key: xx
secret_key: xx
region: us-east-1
discovery:
type: ec2

The same settings I have made on my Data Node instances and have changed only the following parameter:
node.name: DataNode_1
node.master: false
node.data: true

My each service is running perfectly, but when I look at my cluster using ElasticHQ , which I have installed on my Master node , I am able to see mymaster node but I am not able to see my data node.

What may be the problem ?

Thank you

10

Check your ES logs on the other nodes, it should mention something.
Also check your security groups.

11

I have allowed all the traffic for inbound and outbound in my security group.

The following is my log file for Data Node:

[2016-07-06 09:05:55,815][INFO ][node ] [Data Node_1] version[2.3.0], pid[1360], build[8371be8/2016-03-29T07:54:48Z]
[2016-07-06 09:05:55,816][INFO ][node ] [Data Node_1] initializing ...
[2016-07-06 09:05:57,037][INFO ][plugins ] [Data Node_1] modules [reindex, lang-expression, lang-groovy], plugins [cloud-aws], sites []
[2016-07-06 09:05:57,081][INFO ][env ] [Data Node_1] using [1] data paths, mounts [[/ (/dev/xvda1)]], net usable_space [5.6gb], net total_space [7.7gb], spins? [no], types [ext4]
[2016-07-06 09:05:57,081][INFO ][env ] [Data Node_1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2016-07-06 09:05:57,081][WARN ][env ] [Data Node_1] max file descriptors [64000] for elasticsearch process likely too low, consider increasing to at least [65536]
[2016-07-06 09:06:00,590][INFO ][node ] [Data Node_1] initialized
[2016-07-06 09:06:00,593][INFO ][node ] [Data Node_1] starting ...
[2016-07-06 09:06:00,714][INFO ][transport ] [Data Node_1] publish_address {10.0.0.235:9300}, bound_addresses {10.0.0.235:9300}
[2016-07-06 09:06:00,718][INFO ][discovery ] [Data Node_1] mycluster/DRurfcDaS_KgC11ot5Fnkg
[2016-07-06 09:06:30,735][WARN ][discovery ] [Data Node_1] waited for 30s and no initial state was set by the discovery
[2016-07-06 09:06:30,752][INFO ][http ] [Data Node_1] publish_address {10.0.0.235:9200}, bound_addresses {10.0.0.235:9200}
[2016-07-06 09:06:30,752][INFO ][node ] [Data Node_1] started
~

12

You could change the log level for cloud.aws and discovery.ec2 packages to DEBUG and see what it tells.

Please format your logs before posting using </> icon.

13

Thank you sir,

Will take care from next time.

14

Solved.

I don't know how but I rebooted my instance and all is working fine.

One more thing I want to know. Is there any open source notification tool for ES Cluster.

Thank you everyone for response and time.