cont...
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.5.2.jar:6.5.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /127.0.0.1:9301
[2018-12-11T00:57:42,503][TRACE][o.e.d.z.UnicastZenPing ] [ip-172-25-34-206] [1] opening connection to [{[::1]:9300}{2gfRXEeFQaui3c7kJ4lZkQ}{0:0:0:0:0:0:0:1}{[::1]:9300}]
[2018-12-11T00:57:42,506][TRACE][o.e.d.z.UnicastZenPing ] [ip-172-25-34-206] [1] failed to ping {127.0.0.1:9304}{fEHx1WWfSTqFdkOt7_Zrxw}{127.0.0.1}{127.0.0.1:9304}
org.elasticsearch.transport.ConnectTransportException: [][127.0.0.1:9304]
So what am i missing with the EC2 discovery plugin that is causing the cluster to find "localhost" and repeatedly ping it?
It looks like this exact issue was actually broached here, but this thread has no solution.
Here's the (slightly edited) config for this particular master node:
bootstrap.memory_lock: true
node.name: ${HOSTNAME}
action.destructive_requires_name: true
# default is unbounded
indices.fielddata.cache.size: 1%
cluster.name: BlahCluster
discovery.zen.minimum_master_nodes: 2
# only data nodes should have ingest and http capabilities
node.master: true
node.data: false
node.ingest: false
http.enabled: false
xpack.security.enabled: true
xpack.monitoring.enabled: true
path.data: /mnt/blah
path.logs: /mnt/blah
network.host: _ec2:privateIpv4_
plugin.mandatory: discovery-ec2
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery:
zen.hosts_provider: ec2
ec2.groups: sg-BLAH
ec2.host_type: private_ip
ec2.tag.Cluster: SomeClusterTag
ec2.availability_zones: us-west-1b,us-west-1c
ec2.protocol: https
I have confirmed that SomeClusterTag
is on every node I would expect to see in the cluster.
I have confirmed that the security group sg-BLAH
is on every node and allows port 9200 through 9400.
I know that there is no problem with inter-node communications because when I add the hosts manually, the cluster has no problem assembling:
discovery.zen.ping.unicast.hosts: [
"172.25.56.186",
"172.25.48.249",
"172.25.38.47",
"172.25.51.137",
"172.25.34.128"]
So what am i missing?