Datanodes cant see each other using EC2 Discovery Plugin

Elastic Stack Elasticsearch
1

hello guys,

im currently having an issue after configuring the elasticsearch to see the other ec2 datanodes
note that im using elasticsearch-6.0.1-1

the below is my configuration
#####################
cluster.name: test-cluster
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: [eth0,local,ec2]
discovery.ec2.access_key: --
discovery.ec2.secret_key: --
discovery.ec2.availability_zones: eu-west-1c
cloud.node.auto_attributes: true
discovery.ec2.any_group: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
#########

i put datanodes in the same subnet,security groups and zone and they are accessible to each other through 9200-9300

did i miss something?

2

Could you share your logs?
Also you should use the secure settings instead putting in clear your credentials.

3

thank you,

im currently test the creating cluster from scratch,so i should do all running and security in future

this is the log

[2017-12-16T12:07:20,499][INFO ][o.e.d.DiscoveryModule    ] [ip-10-10-227-107] using discovery type [zen]
[2017-12-16T12:07:20,954][INFO ][o.e.n.Node               ] [ip-10-10-227-107] initialized
[2017-12-16T12:07:20,954][INFO ][o.e.n.Node               ] [ip-10-10-227-107] starting ...
[2017-12-16T12:07:21,101][INFO ][o.e.t.TransportService   ] [ip-10-10-227-107] publish_address {10.10.227.107:9300}, bound_addresses {10.10.227.107:9300}, {[::1]:9300}, {127.0.0.1:9300}, {[fe80::4d0:79ff:fe62:a2fa]:9300}
[2017-12-16T12:07:21,116][INFO ][o.e.b.BootstrapChecks    ] [ip-10-10-227-107] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-12-16T12:07:25,108][INFO ][o.e.c.s.MasterService    ] [ip-10-10-227-107] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {ip-10-10-227-107}{VzczLDl2Sqi_GoGafJA8fg}{eI97wxT5TzauC6ZVOr3QIQ}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c}
[2017-12-16T12:07:25,119][INFO ][o.e.c.s.ClusterApplierService] [ip-10-10-227-107] new_master {ip-10-10-227-107}{VzczLDl2Sqi_GoGafJA8fg}{eI97wxT5TzauC6ZVOr3QIQ}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c}, reason: apply cluster state (from master [master {ip-10-10-227-107}{VzczLDl2Sqi_GoGafJA8fg}{eI97wxT5TzauC6ZVOr3QIQ}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2017-12-16T12:07:25,156][INFO ][o.e.g.GatewayService     ] [ip-10-10-227-107] recovered [0] indices into cluster_state
[2017-12-16T12:07:25,183][INFO ][o.e.h.n.Netty4HttpServerTransport] [ip-10-10-227-107] publish_address {10.10.227.107:9200}, bound_addresses {10.10.227.107:9200}, {[::1]:9200}, {127.0.0.1:9200}, {[fe80::4d0:79ff:fe62:a2fa]:9200}
[2017-12-16T12:07:25,183][INFO ][o.e.n.Node               ] [ip-10-10-227-107] started
4

Please format your code using </> icon as explained in this guide. It will make your post more readable.

Or use markdown style like:

```
CODE
```

What are the other nodes logs?

5

changed:+1:

i have two nodes for now,,

this is the other node

12-16T12:07:15,182][INFO ][o.e.d.DiscoveryModule    ] [ip-10-10-227-55] using discovery type [zen]
[2017-12-16T12:07:15,649][INFO ][o.e.n.Node               ] [ip-10-10-227-55] initialized
[2017-12-16T12:07:15,650][INFO ][o.e.n.Node               ] [ip-10-10-227-55] starting ...
[2017-12-16T12:07:15,805][INFO ][o.e.t.TransportService   ] [ip-10-10-227-55] publish_address {10.10.227.55:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300}, {[fe80::404:f6ff:fe0d:8284]:9300}, {10.10.227.55:9300}
[2017-12-16T12:07:15,816][INFO ][o.e.b.BootstrapChecks    ] [ip-10-10-227-55] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-12-16T12:07:19,859][INFO ][o.e.c.s.MasterService    ] [ip-10-10-227-55] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {ip-10-10-227-55}{xOQmeyPGSMK3whAd37MKXw}{WHgsFN4CS7-lwynYDPRK8Q}{10.10.227.55}{10.10.227.55:9300}{aws_availability_zone=eu-west-1c}
[2017-12-16T12:07:19,866][INFO ][o.e.c.s.ClusterApplierService] [ip-10-10-227-55] new_master {ip-10-10-227-55}{xOQmeyPGSMK3whAd37MKXw}{WHgsFN4CS7-lwynYDPRK8Q}{10.10.227.55}{10.10.227.55:9300}{aws_availability_zone=eu-west-1c}, reason: apply cluster state (from master [master {ip-10-10-227-55}{xOQmeyPGSMK3whAd37MKXw}{WHgsFN4CS7-lwynYDPRK8Q}{10.10.227.55}{10.10.227.55:9300}{aws_availability_zone=eu-west-1c} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2017-12-16T12:07:19,893][INFO ][o.e.g.GatewayService     ] [ip-10-10-227-55] recovered [0] indices into cluster_state
[2017-12-16T12:07:19,907][INFO ][o.e.h.n.Netty4HttpServerTransport] [ip-10-10-227-55] publish_address {10.10.227.55:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200}, {[fe80::404:f6ff:fe0d:8284]:9200}, {10.10.227.55:9200}
[2017-12-16T12:07:19,907][INFO ][o.e.n.Node               ] [ip-10-10-227-55] started

thank you

6

You need to change some settings.

Please read: https://www.elastic.co/guide/en/elasticsearch/plugins/current/discovery-ec2-usage.html

7

do you mean

discovery.zen.hosts_provider: ec2

if yes sorry i already added this to the configuration and added also

discovery.ec2.host_type: private_ip

but with no use.maybe i will dig this issue later

thanks

8

hello again,

actually i went through the documentation,made some changes ,but still cant let the two nodes see each other

cluster.name: dev-cluster
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: eu-west-1c
cloud.node.auto_attributes: true
discovery.ec2.groups: sg-xx,sg-xx
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.Name: Elasticsearch6
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: eu-west-1

the log stayed the same

[2017-12-17T16:22:19,041][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [aggs-matrix-stats]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [analysis-common]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [ingest-common]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [lang-expression]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [lang-mustache]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [lang-painless]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [parent-join]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [percolator]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [reindex]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [repository-url]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [transport-netty4]
[2017-12-17T16:22:19,042][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded module [tribe]
[2017-12-17T16:22:19,049][INFO ][o.e.p.PluginsService     ] [ip-10-10-227-107] loaded plugin [discovery-ec2]
[2017-12-17T16:22:21,040][INFO ][o.e.d.DiscoveryModule    ] [ip-10-10-227-107] using discovery type [zen]
[2017-12-17T16:22:21,510][INFO ][o.e.n.Node               ] [ip-10-10-227-107] initialized
[2017-12-17T16:22:21,510][INFO ][o.e.n.Node               ] [ip-10-10-227-107] starting ...
[2017-12-17T16:22:21,660][INFO ][o.e.t.TransportService   ] [ip-10-10-227-107] publish_address {10.10.227.107:9300}, bound_addresses {10.10.227.107:9300}, {[::1]:9300}, {127.0.0.1:9300}, {[fe80::4d0:79ff:fe62:a2fa]:9300}
[2017-12-17T16:22:21,670][INFO ][o.e.b.BootstrapChecks    ] [ip-10-10-227-107] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-12-17T16:22:21,863][INFO ][o.e.d.e.AwsEc2UnicastHostsProvider] [ip-10-10-227-107] Exception while retrieving instance list from AWS API: Unable to execute HTTP request: eu-west-1: Name or service not known
[2017-12-17T16:22:24,904][INFO ][o.e.c.s.MasterService    ] [ip-10-10-227-107] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {ip-10-10-227-107}{GAri8yyrReSaR24rbU-zeQ}{8UjTAr2FT-6GpIB7oQ7Q9Q}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c}
[2017-12-17T16:22:24,910][INFO ][o.e.c.s.ClusterApplierService] [ip-10-10-227-107] new_master {ip-10-10-227-107}{GAri8yyrReSaR24rbU-zeQ}{8UjTAr2FT-6GpIB7oQ7Q9Q}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c}, reason: apply cluster state (from master [master {ip-10-10-227-107}{GAri8yyrReSaR24rbU-zeQ}{8UjTAr2FT-6GpIB7oQ7Q9Q}{10.10.227.107}{10.10.227.107:9300}{aws_availability_zone=eu-west-1c} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2017-12-17T16:22:24,943][INFO ][o.e.g.GatewayService     ] [ip-10-10-227-107] recovered [0] indices into cluster_state
[2017-12-17T16:22:24,950][INFO ][o.e.h.n.Netty4HttpServerTransport] [ip-10-10-227-107] publish_address {10.10.227.107:9200}, bound_addresses {10.10.227.107:9200}, {[::1]:9200}, {127.0.0.1:9200}, {[fe80::4d0:79ff:fe62:a2fa]:9200}
[2017-12-17T16:22:24,950][INFO ][o.e.n.Node               ] [ip-10-10-227-107] started
[2017-12-17T16:22:54,943][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:23:24,946][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:23:54,947][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:24:24,948][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:24:54,949][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:25:24,950][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:25:54,951][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:26:24,953][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[2017-12-17T16:26:48,805][TRACE][o.e.a.a.c.s.TransportClusterStateAction] [ip-10-10-227-107] Serving cluster state request using version 2
[2017-12-17T16:26:54,954][TRACE][o.e.a.a.i.s.TransportIndicesStatsAction] [ip-10-10-227-107] resolving shards for [indices:monitor/stats] based on cluster state version [2]
[root@ip-10-10-227-107 elasticsearch]#

i dont know if there any further configuration,but i covered all the configuration for ec2 discovery,any suggestions?

thank you

9

Can you telnet from one node to the other on ports 9200 and 9300?

10

yes i can telnet without issues

11

Try using _publicIp_ for discovery! :frowning:

12

@mujtabahussain it didnt work as i dont have public IP in any instance :frowning:

im going insane because of that :open_mouth:

13

DOH. So Sorry, I meant _privateIp_ not public :frowning:

14

nothing worked :/.

did anyone try it on the new version of elasticsearch

15

You need to change I think:

discovery.ec2.endpoint: eu-west-1

By

discovery.ec2.endpoint: ec2.eu-west-1.amazonaws.com
3 Likes
16

@dadoonet u r the man,it worked :smiley: :smiley:
the configuration should be like the below

cluster.name: dev-cluster
node.name: ${HOSTNAME}
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: [_eth0_,_local_,_ec2_]
discovery.zen.hosts_provider: ec2
discovery.ec2.availability_zones: eu-west-1c,eu-west-1b,eu-west-1a
cloud.node.auto_attributes: true
#discovery.ec2.groups: sg-xx,sg-xx
discovery.ec2.any_group: true
discovery.ec2.host_type: private_ip
discovery.ec2.tag.Name: Elasticsearch6
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery.ec2.endpoint: ec2.eu-west-1.amazonaws.com
discovery.zen.minimum_master_nodes: 2
2 Likes
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.