"not enough master nodes discovered during pinging" Error in AWS WEST only East comes up fine!

Elastic Stack Elasticsearch
1

I understand this has been asked before. Our situation is a bit odd where this USED to work a few days ago and does work on our AWS EAST stack. However, we get this error on our WEST region stack. The security groups haven't changed (used to work with those). Both east and west were recently upgraded to use 6.3.0. East came up fine and the security groups between east and west look consistent. The SAME security groups worked a few days ago in 5.5 version of the stack. Somehow after upgrading to 6.3.0, east came up perfectly but west gives the
"not enough master nodes discovered during pinging" error during bootstrap. Any suggestions?

Elastic YML

cluster.name: my-cluster
node.data: false
node.master: false
node.ingest: false
search.remote.connect: false
node.name: ${HOSTNAME}
path.data: /opt/mount/var/data/my-cluster/elasticsearch
path.logs: /opt/mount/var/log/my-cluster/elasticsearch
network.host: _ec2:privateIpv4_
discovery.zen.hosts_provider: ec2
discovery.zen.minimum_master_nodes: 3
discovery.ec2.host_type: private_ip
discovery.ec2.tag.Name: my-cluster-tag
discovery.ec2.groups: our-west-sg // EAST SG WORKS. THIS IS WEST. RULES ARE SIMILAR
discovery.ec2.any_group: false
discovery.ec2.proxy.host: ourproxy
discovery.ec2.proxy.port: 8099
cloud.node.auto_attributes: true
action.destructive_requires_name: true
bootstrap.memory_lock: true
http.type: ssl_netty4
xpack.security.enabled: false
2

Is this on the AWS Elasticsearch service?

3

no - 6.3.0 installed on AWS EC2

4

FYI - I do see this in the logs in the WEST. But I don't have the localhost (127.0.0.1) configured anywhere!

[2018-07-11T04:02:13,828][TRACE][o.e.d.z.UnicastZenPing   ] [ip-10-184-180-157.myhost.com] [14] failed to ping {#zen_unicast_127.0.0.1_1#}{1wrYu3e5Qq2FFZRbPn8npw}{127.0.0.1}{12
        7.0.0.1:9301}
  19564 org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9301] connect_exception
  19565         at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.3.0.jar:6.3.0]
  19566         at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:631) ~[elasticsearch-6.3.0.jar:6.3.0]
  19567         at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:123) ~[elasticsearch-6.3.0.jar:6.3.0]
  19568         at org.elasticsearch.transport.TransportService.openConnection(TransportService.java:350) ~[elasticsearch-6.3.0.jar:6.3.0]
  19569         at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:397) ~[elasticsearch-6.3.0.jar:6.3.0]
  19570         at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:502) [elasticsearch-6.3.0.jar:6.3.0]
  19571         at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:724) [elasticsearch-6.3.0.jar:6.3.0]
  19572         at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
  19573         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
  19574         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
  19575         at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]

and this

org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9300] connect_exception
        at org.elasticsearch.transport.TcpChannel.awaitConnected(TcpChannel.java:165) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:631) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:123) ~[elasticsearch-6.3.0.jar:6.3.0]
5

Found the problem.

Discovery EC2 needs Needs an explicit endpoint added to point to that region. Reason why it worked in the EAST was it looks like US EAST 1 seems to be implicit in EC2 instances brought up in EAST

Adding this to YML fixed it! Note to Devs - Perhaps it might be prudent to add this as a message in INFO as a suggestive tip on discovery-ec2 bootstrapping? (something like a hint? ). Hope this helps anyone googling this.

discovery.ec2.endpoint: ec2.us-west-2.amazonaws.com

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.