Elasticsearch on AWS ECS - Master node itself unable to discover master

Hi I am trying to run elasticsearch on ECS.

I am using simple elasticsearch.yml as below for the docker I am using for the Master Service.

=================== Start Elasticsearch YML ====================

cluster.name: "ELKecs"
node.name: mst-node-1
cluster.initial_master_nodes:
- mst-node-1
- mst-node-2
discovery.seed_providers: ec2
network.host: 0.0.0.0
network.publish_host: ec2
transport.publish_host: ec2
transport.port: 9300
http.port: 9200
discovery.ec2.tag.role: master
discovery.ec2.tag.environment: dev,staging
discovery.ec2.tag.ElasticSearch: "poc"
discovery.ec2.endpoint: ec2.{REGION}.amazonaws.com s3.client.default.endpoint: s3.{REGION}.amazonaws.com
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
xpack.security.enabled: false

=================== End Elasticsearch YML ====================

but this results to an error, as below... and I wonder why its so.

{
"type": "server",
"timestamp": "2020-11-20T08:48:18,089Z",
"level": "WARN",
"component": "o.e.c.c.ClusterFormationFailureHelper",
"cluster.name": "ELKecs",
"node.name": "mst-node-1",
"message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [mst-node-1, mst-node-2] to bootstrap a cluster: have discovered [{mst-node-1}{nEIZsF-kTEetq2duqvW3aA}{tSlxUtHSRteDzE6gekNUWg}{10.44.41.150}{10.44.41.150:9300}{cdhilmrstw}{aws_availability_zone=ap-southeast-1b, ml.machine_memory=4294967296, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300] from hosts providers and [{mst-node-1}{nEIZsF-kTEetq2duqvW3aA}{tSlxUtHSRteDzE6gekNUWg}{10.44.41.150}{10.44.41.150:9300}{cdhilmrstw}{aws_availability_zone=ap-southeast-1b, ml.machine_memory=4294967296, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"
}

Hi, I am having the same problem but on a different platform I am running on a private cloud provider but I think the problem is here..

The hostname that elastic is discovering is all that and in your case and mine it should be only the hostname, but I do not know how to change that

The message says all we need, although it is quite verbose. Here's the key part:

So it hasn't discovered mst-node-2, without which it can't proceed. This is because ...

... it's only looking on localhost. You're trying to use discovery-ec2 but there's something wrong with its setup so it is not discovering the address of the other node.

Hi David, here is my error message:

es02    | {"type": "server", "timestamp": "2020-12-03T14:42:57,800Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es01, es02, es03] to bootstrap a cluster: have discovered [{es02}{YJiCiKnMSnuwqWrQGF0ojg}{OPwfpbcmQuicOGrm6aR-tg}{172.20.0.2}{172.20.0.2:9300}{dilmrt}{ml.machine_memory=16643383296, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.20.0.3:9300, 172.20.0.4:9300] from hosts providers and [{es02}{YJiCiKnMSnuwqWrQGF0ojg}{OPwfpbcmQuicOGrm6aR-tg}{172.20.0.2}{172.20.0.2:9300}{dilmrt}{ml.machine_memory=16643383296, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

I am traying to use docker on centos 8 , the docker-compose file I'm using is this one:

Same as above: it's looking for multiple nodes but only discovering one:

The addresses look reasonable here, they're not just localhost, so in your case I'd guess it's a connectivity issue.

Thanks for your reply David,

...but now I am confused, I am running this on a single node via docker, iptables has no rules and selinux is Permisive, how can this be a connectivity issue ?

Sorry I've no idea, it says it's looking for the other nodes at those addresses, but it's not discovering them. Perhaps there's other log messages indicating why; you could try setting logger.org.elasticsearch.discovery: DEBUG, perhaps that'll show some useful exceptions happening.

Ready.

Disable firewalld, restart docker and done