Node Discovery in elasticsearch on amazon EC2


(Siddharth Gupta) #1

Hello,

I was using elastic search 1.7.0 on amzon EC2 and everything was working fine.

The nodes were able to discover each other and joined together.

But when I switched to elasticsearch 2.1.0 things have been difficult.

The nodes are not able to discover each other.

Have they made any difference ?
Please help me with this issue.

Thanks & regards
Siddharth


(David Pilato) #2

Probably network.host not set.


(Vlad Miller) #3

Perhaps there could be few issues.

  • Make sure your ES nodes listen on correct interface
  • Make sure your security groups are open and set to allow 9300-9400 traffic between nodes
  • Use aws-ec2 discovery plugin to discover other nodes

Also check error logs to see if there any useful information


(Siddharth Gupta) #4

Thanks for your reply!
When i set
**network.host : ec2:publicIpv4 **
the publish address is same as the IP of my instance but when I start the nodes it says bound address could not be assigned.
Do you know what can be the missing part ?

Thanks & regards
Siddharth Gupta


(Siddharth Gupta) #5

@vladmiller Thanks for the reply.
I am sure about the last two points in your reply.
Can you please elaborate a little about the first one ?
Error logs do not show any useful information.

Thanks & Regards
Siddharth Gupta.


(David Pilato) #6

Can you share your logs?


(Vlad Miller) #7

Here is my config. I am using cloud-aws plugin to discover other nodes

Sorry, cannot paste it properly here
http://pastebin.com/e8Ert1h7


(Siddharth Gupta) #8

@dadoonet
[m] publish_address {52.23.160.86:9300}, bound_addresses {127.0.0.1:9300}, {[::1]:9300}

where 'm' is the name of my node. publish address is fine but I think there is some problem with bound_address. Am I right ?

When I use network.host: ec2:publicIpv4 it says:

Exception in thread "main" BindTransportException[Failed to bind to [9300-9400]]; nested: ChannelException[Failed to bind to: /52.23.160.86:9400]; nested: BindException[Cannot assign requested address];
Likely root cause: java.net.BindException: Cannot assign requested address

Can you see something through it ?

Thanks & Regards
Sidddharth Gupta


(Siddharth Gupta) #9

Thanks & regards
Siddharth


(Siddharth Gupta) #10

I used the same config on elasticsearch 2.1.0
It is working fine in elasticsearch 1.7.1


(Siddharth Gupta) #11

When I start the other node it says::

[2015-12-08 09:18:16,192][WARN ][discovery ] [m+d] waited for 30s and no initial state was set by the discovery
[2015-12-08 09:18:16,272][INFO ][http ] [m+d] publish_address {52.23.160.86:9201}, bound_addresses {127.0.0.1:9201}, {[::1]:9201}
[2015-12-08 09:18:16,274][INFO ][node ] [m+d] started

That means the node has started but the node has not joined the old one.


(David Pilato) #12

Your config is wrong for sure. May be share it with us?


(Siddharth Gupta) #15
cluster.name: my-app
node.name: m+d
node.data: true
node.master: true
bootstrap.mlockall: true
network.publish_host: _ec2:publicIpv4_ 
discovery.ec2.host_type: public_dns
cloud.aws.access_key: XXXXXXXXXXXXXXXXXXXXXXX
cloud.aws.secret_key: XXXXXXXXXXXXXXXXXXXXXXX
discovery.zen.ping.timeout: 30s
discovery.zen.ping.multicast.enabled: false
discovery.type: ec2
discovery.zen.ping.unicast.hosts: ["ec2-52-23-160-86.compute-1.amazonaws.com"]
discovery.zen.minimum_master_nodes: 2
http.cors.enabled: true
http.cors.allow-origin: "*"

(Siddharth Gupta) #16

Above is my elasticsearch.yml

Can you point out where I am messing things up

Thanks
Siddharth


(David Pilato) #17

I edited your answer to shows only uncommented settings.

Some comments:

  • Set network.publish_host: _ec2_
  • Remove network.publish_host: _ec2:publicIpv4_
  • Remove discovery.ec2.host_type: public_dns
  • Remove discovery.zen.ping.unicast.hosts: ["ec2-52-23-160-86.compute-1.amazonaws.com"]
  • Remove discovery.zen.ping.multicast.enabled: false

And restart...

So basically try elasticsearch.yml:

cluster.name: my-app
node.name: m+d
bootstrap.mlockall: true
network.host: _ec2_ 
cloud.aws.access_key: XXXXXXXXXXXXXXXXXXXXXXX
cloud.aws.secret_key: XXXXXXXXXXXXXXXXXXXXXXX
discovery.type: ec2
discovery.zen.minimum_master_nodes: 2

Let me know how it goes.


Can't get ec2 discovery working
(Siddharth Gupta) #18

That worked ! Thanks a lot!


(Python Coder) #19

gorgeous :wink:


(system) #20