Datanodes cant see each other using EC2 Discovery Plugin

hello guys,

im currently having an issue after configuring the elasticsearch to see the other ec2 datanodes
note that im using elasticsearch-6.0.1-1

the below is my configuration
#####################
cluster.name: test-cluster
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: [eth0,local,ec2]
discovery.ec2.access_key: --
discovery.ec2.secret_key: --
discovery.ec2.availability_zones: eu-west-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
#########

i put datanodes in the same subnet,security groups and zone and they are accessible to each other through 9200-9300

did i miss something?

Could you share your logs?
Also you should use the secure settings instead putting in clear your credentials.

thank you,

im currently test the creating cluster from scratch,so i should do all running and security in future

this is the log

[2017-12-16T12:07:20,499][INFO ][o.e.d.DiscoveryModule    ] [ip-10-10-227-107] using discovery type [zen]
[2017-12-16T12:07:20,954][INFO ][o.e.n.Node               ] [ip-10-10-227-107] initialized
[2017-12-16T12:07:20,954][INFO ][o.e.n.Node               ] [ip-10-10-227-107] starting ...
[2017-12-16T12:07:21,101][INFO ][o.e.t.TransportService   ] [ip-10-10-227-107] publish_address {10.10.227.107:9300}, bound_addresses {10.10.227.107:9300}, {[::1]:9300}, {127.0.0.1:9300}, {[fe80::4d0:79ff:fe62:a2fa]:9300}
[2017-12-16T12:07:21,116][INFO ][o.e.b.BootstrapChecks    ] [ip-10-10-227-107] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-12-16T12:07:25,108][INFO ][o.e.c.s.MasterService    ] [ip-10-10-227-107] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {ip-10-10-227-107}{VzczLDl2Sqi_GoGafJA8fg}{eI97wxT5TzauC6ZVOr3QIQ}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c}
[2017-12-16T12:07:25,119][INFO ][o.e.c.s.ClusterApplierService] [ip-10-10-227-107] new_master {ip-10-10-227-107}{VzczLDl2Sqi_GoGafJA8fg}{eI97wxT5TzauC6ZVOr3QIQ}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c}, reason: apply cluster state (from master [master {ip-10-10-227-107}{VzczLDl2Sqi_GoGafJA8fg}{eI97wxT5TzauC6ZVOr3QIQ}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2017-12-16T12:07:25,156][INFO ][o.e.g.GatewayService     ] [ip-10-10-227-107] recovered [0] indices into cluster_state
[2017-12-16T12:07:25,183][INFO ][o.e.h.n.Netty4HttpServerTransport] [ip-10-10-227-107] publish_address {10.10.227.107:9200}, bound_addresses {10.10.227.107:9200}, {[::1]:9200}, {127.0.0.1:9200}, {[fe80::4d0:79ff:fe62:a2fa]:9200}
[2017-12-16T12:07:25,183][INFO ][o.e.n.Node               ] [ip-10-10-227-107] started

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

What are the other nodes logs?

changed:+1:

i have two nodes for now,,

this is the other node

12-16T12:07:15,182][INFO ][o.e.d.DiscoveryModule    ] [ip-10-10-227-55] using discovery type [zen]
[2017-12-16T12:07:15,649][INFO ][o.e.n.Node               ] [ip-10-10-227-55] initialized
[2017-12-16T12:07:15,650][INFO ][o.e.n.Node               ] [ip-10-10-227-55] starting ...
[2017-12-16T12:07:15,805][INFO ][o.e.t.TransportService   ] [ip-10-10-227-55] publish_address {10.10.227.55:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {[fe80::404:f6ff:fe0d:8284]:9300}, {10.10.227.55:9300}
[2017-12-16T12:07:15,816][INFO ][o.e.b.BootstrapChecks    ] [ip-10-10-227-55] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-12-16T12:07:19,859][INFO ][o.e.c.s.MasterService    ] [ip-10-10-227-55] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {ip-10-10-227-55}{xOQmeyPGSMK3whAd37MKXw}{WHgsFN4CS7-lwynYDPRK8Q}{10.10.227.55}{10.10.227.55:9300}{aws_availability_zone=eu-west-1c}
[2017-12-16T12:07:19,866][INFO ][o.e.c.s.ClusterApplierService] [ip-10-10-227-55] new_master {ip-10-10-227-55}{xOQmeyPGSMK3whAd37MKXw}{WHgsFN4CS7-lwynYDPRK8Q}{10.10.227.55}{10.10.227.55:9300}{aws_availability_zone=eu-west-1c}, reason: apply cluster state (from master [master {ip-10-10-227-55}{xOQmeyPGSMK3whAd37MKXw}{WHgsFN4CS7-lwynYDPRK8Q}{10.10.227.55}{10.10.227.55:9300}{aws_availability_zone=eu-west-1c} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2017-12-16T12:07:19,893][INFO ][o.e.g.GatewayService     ] [ip-10-10-227-55] recovered [0] indices into cluster_state
[2017-12-16T12:07:19,907][INFO ][o.e.h.n.Netty4HttpServerTransport] [ip-10-10-227-55] publish_address {10.10.227.55:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {[fe80::404:f6ff:fe0d:8284]:9200}, {10.10.227.55:9200}
[2017-12-16T12:07:19,907][INFO ][o.e.n.Node               ] [ip-10-10-227-55] started

thank you

You need to change some settings.

Please read: https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-ec2-usage.html

do you mean

discovery.zen.hosts_provider: ec2

if yes sorry i already added this to the configuration and added also

discovery.ec2.host_type: private_ip

but with no use.maybe i will dig this issue later

thanks

hello again,

actually i went through the documentation,made some changes ,but still cant let the two nodes see each other

cluster.name: dev-cluster
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: eu-west-1c
cloud.node.auto_attributes: true
discovery.ec2.groups: sg-xx,sg-xx
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.Name: Elasticsearch6
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: eu-west-1

the log stayed the same

[2017-12-17T16:22:19,041][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [aggs-matrix-stats]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [analysis-common]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [ingest-common]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [lang-expression]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [lang-mustache]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [lang-painless]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [parent-join]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [percolator]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [reindex]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [repository-url]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [transport-netty4]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [tribe]
[2017-12-17T16:22:19,049][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded plugin [discovery-ec2]
[2017-12-17T16:22:21,040][INFO ][o.e.d.DiscoveryModule    ] [ip-10-10-227-107] using discovery type [zen]
[2017-12-17T16:22:21,510][INFO ][o.e.n.Node               ] [ip-10-10-227-107] initialized
[2017-12-17T16:22:21,510][INFO ][o.e.n.Node               ] [ip-10-10-227-107] starting ...
[2017-12-17T16:22:21,660][INFO ][o.e.t.TransportService   ] [ip-10-10-227-107] publish_address {10.10.227.107:9300}, bound_addresses {10.10.227.107:9300}, {[::1]:9300}, {127.0.0.1:9300}, {[fe80::4d0:79ff:fe62:a2fa]:9300}
[2017-12-17T16:22:21,670][INFO ][o.e.b.BootstrapChecks    ] [ip-10-10-227-107] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-12-17T16:22:21,863][INFO ][o.e.d.e.AwsEc2UnicastHostsProvider] [ip-10-10-227-107] Exception while retrieving instance list from AWS API: Unable to execute HTTP request: eu-west-1: Name or service not known
[2017-12-17T16:22:24,904][INFO ][o.e.c.s.MasterService    ] [ip-10-10-227-107] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {ip-10-10-227-107}{GAri8yyrReSaR24rbU-zeQ}{8UjTAr2FT-6GpIB7oQ7Q9Q}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c}
[2017-12-17T16:22:24,910][INFO ][o.e.c.s.ClusterApplierService] [ip-10-10-227-107] new_master {ip-10-10-227-107}{GAri8yyrReSaR24rbU-zeQ}{8UjTAr2FT-6GpIB7oQ7Q9Q}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c}, reason: apply cluster state (from master [master {ip-10-10-227-107}{GAri8yyrReSaR24rbU-zeQ}{8UjTAr2FT-6GpIB7oQ7Q9Q}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2017-12-17T16:22:24,943][INFO ][o.e.g.GatewayService     ] [ip-10-10-227-107] recovered [0] indices into cluster_state
[2017-12-17T16:22:24,950][INFO ][o.e.h.n.Netty4HttpServerTransport] [ip-10-10-227-107] publish_address {10.10.227.107:9200}, bound_addresses {10.10.227.107:9200}, {[::1]:9200}, {127.0.0.1:9200}, {[fe80::4d0:79ff:fe62:a2fa]:9200}
[2017-12-17T16:22:24,950][INFO ][o.e.n.Node               ] [ip-10-10-227-107] started
[2017-12-17T16:22:54,943][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:23:24,946][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:23:54,947][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:24:24,948][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:24:54,949][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:25:24,950][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:25:54,951][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:26:24,953][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:26:48,805][TRACE][o.e.a.a.c.s.TransportClusterStateAction] [ip-10-10-227-107] Serving cluster state request using version 2
[2017-12-17T16:26:54,954][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[root@ip-10-10-227-107 elasticsearch]# 

i dont know if there any further configuration,but i covered all the configuration for ec2 discovery,any suggestions?

thank you

Can you telnet from one node to the other on ports 9200 and 9300?

yes i can telnet without issues

Try using _publicIp_ for discovery! :frowning:

@mujtabahussain it didnt work as i dont have public IP in any instance :frowning:

im going insane because of that :open_mouth:

DOH. So Sorry, I meant _privateIp_ not public :frowning:

nothing worked :/.

did anyone try it on the new version of elasticsearch

You need to change I think:

discovery.ec2.endpoint: eu-west-1

By

discovery.ec2.endpoint: ec2.eu-west-1.amazonaws.com
3 Likes

@dadoonet u r the man,it worked :smiley: :smiley:
the configuration should be like the below

cluster.name: dev-cluster
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: eu-west-1c,eu-west-1b,eu-west-1a
cloud.node.auto_attributes: true
#discovery.ec2.groups: sg-xx,sg-xx
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.Name: Elasticsearch6
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.eu-west-1.amazonaws.com
discovery.zen.minimum_master_nodes: 2
2 Likes

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.