Node Discovery in elasticsearch on amazon EC2

Elastic Stack Elasticsearch
1

Hello,

I was using elastic search 1.7.0 on amzon EC2 and everything was working fine.

The nodes were able to discover each other and joined together.

But when I switched to elasticsearch 2.1.0 things have been difficult.

The nodes are not able to discover each other.

Have they made any difference ?
Please help me with this issue.

Thanks & regards
Siddharth

2

Probably network.host not set.

3

Perhaps there could be few issues.

  • Make sure your ES nodes listen on correct interface
  • Make sure your security groups are open and set to allow 9300-9400 traffic between nodes
  • Use aws-ec2 discovery plugin to discover other nodes

Also check error logs to see if there any useful information

4

Thanks for your reply!
When i set
**network.host : ec2:publicIpv4 **
the publish address is same as the IP of my instance but when I start the nodes it says bound address could not be assigned.
Do you know what can be the missing part ?

Thanks & regards
Siddharth Gupta

5

@vladmiller Thanks for the reply.
I am sure about the last two points in your reply.
Can you please elaborate a little about the first one ?
Error logs do not show any useful information.

Thanks & Regards
Siddharth Gupta.

6

Can you share your logs?

7

Here is my config. I am using cloud-aws plugin to discover other nodes

Sorry, cannot paste it properly here
http://pastebin.com/e8Ert1h7

8

@dadoonet
[m] publish_address {52.23.160.86:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}

where 'm' is the name of my node. publish address is fine but I think there is some problem with bound_address. Am I right ?

When I use network.host: ec2:publicIpv4 it says:

Exception in thread "main" BindTransportException[Failed to bind to [9300-9400]]; nested: ChannelException[Failed to bind to: /52.23.160.86:9400]; nested: BindException[Cannot assign requested address];
Likely root cause: java.net.BindException: Cannot assign requested address

Can you see something through it ?

Thanks & Regards
Sidddharth Gupta

9

Thanks & regards
Siddharth

10

I used the same config on elasticsearch 2.1.0
It is working fine in elasticsearch 1.7.1

11

When I start the other node it says::

[2015-12-08 09:18:16,192][WARN ][discovery ] [m+d] waited for 30s and no initial state was set by the discovery
[2015-12-08 09:18:16,272][INFO ][http ] [m+d] publish_address {52.23.160.86:9201}, bound_addresses {127.0.0.1:9201}, {[::1]:9201}
[2015-12-08 09:18:16,274][INFO ][node ] [m+d] started

That means the node has started but the node has not joined the old one.

12

Your config is wrong for sure. May be share it with us?

15 
cluster.name: my-app
node.name: m+d
node.data: true
node.master: true
bootstrap.mlockall: true
network.publish_host: _ec2:publicIpv4_ 
discovery.ec2.host_type: public_dns
cloud.aws.access_key: XXXXXXXXXXXXXXXXXXXXXXX
cloud.aws.secret_key: XXXXXXXXXXXXXXXXXXXXXXX
discovery.zen.ping.timeout: 30s
discovery.zen.ping.multicast.enabled: false
discovery.type: ec2
discovery.zen.ping.unicast.hosts: ["ec2-52-23-160-86.compute-1.amazonaws.com"]
discovery.zen.minimum_master_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"
16

Above is my elasticsearch.yml

Can you point out where I am messing things up

Thanks
Siddharth

17

I edited your answer to shows only uncommented settings.

Some comments:

  • Set network.publish_host: _ec2_
  • Remove network.publish_host: _ec2:publicIpv4_
  • Remove discovery.ec2.host_type: public_dns
  • Remove discovery.zen.ping.unicast.hosts: ["ec2-52-23-160-86.compute-1.amazonaws.com"]
  • Remove discovery.zen.ping.multicast.enabled: false

And restart...

So basically try elasticsearch.yml:

cluster.name: my-app
node.name: m+d
bootstrap.mlockall: true
network.host: _ec2_ 
cloud.aws.access_key: XXXXXXXXXXXXXXXXXXXXXXX
cloud.aws.secret_key: XXXXXXXXXXXXXXXXXXXXXXX
discovery.type: ec2
discovery.zen.minimum_master_nodes: 2

Let me know how it goes.

1 Like
18

That worked ! Thanks a lot!

19

gorgeous :wink: