Elasticsearch on AWS ECS - Master node itself unable to discover master

Elastic Stack Elasticsearch
1

Hi I am trying to run elasticsearch on ECS.

I am using simple elasticsearch.yml as below for the docker I am using for the Master Service.

=================== Start Elasticsearch YML ====================

cluster.name: "ELKecs"
node.name: mst-node-1
cluster.initial_master_nodes:
- mst-node-1
- mst-node-2
discovery.seed_providers: ec2
network.host: 0.0.0.0
network.publish_host: ec2
transport.publish_host: ec2
transport.port: 9300
http.port: 9200
discovery.ec2.tag.role: master
discovery.ec2.tag.environment: dev,staging
discovery.ec2.tag.ElasticSearch: "poc"
discovery.ec2.endpoint: ec2.{REGION}.amazonaws.com s3.client.default.endpoint: s3.{REGION}.amazonaws.com
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
xpack.security.enabled: false

=================== End Elasticsearch YML ====================

but this results to an error, as below... and I wonder why its so.

{
"type": "server",
"timestamp": "2020-11-20T08:48:18,089Z",
"level": "WARN",
"component": "o.e.c.c.ClusterFormationFailureHelper",
"cluster.name": "ELKecs",
"node.name": "mst-node-1",
"message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [mst-node-1, mst-node-2] to bootstrap a cluster: have discovered [{mst-node-1}{nEIZsF-kTEetq2duqvW3aA}{tSlxUtHSRteDzE6gekNUWg}{10.44.41.150}{10.44.41.150:9300}{cdhilmrstw}{aws_availability_zone=ap-southeast-1b, ml.machine_memory=4294967296, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [127.0.0.1:9300] from hosts providers and [{mst-node-1}{nEIZsF-kTEetq2duqvW3aA}{tSlxUtHSRteDzE6gekNUWg}{10.44.41.150}{10.44.41.150:9300}{cdhilmrstw}{aws_availability_zone=ap-southeast-1b, ml.machine_memory=4294967296, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0"
}

2

Hi, I am having the same problem but on a different platform I am running on a private cloud provider but I think the problem is here..

The hostname that elastic is discovering is all that and in your case and mine it should be only the hostname, but I do not know how to change that

3

The message says all we need, although it is quite verbose. Here's the key part:

So it hasn't discovered mst-node-2, without which it can't proceed. This is because ...

... it's only looking on localhost. You're trying to use discovery-ec2 but there's something wrong with its setup so it is not discovering the address of the other node.

4

Hi David, here is my error message:

es02    | {"type": "server", "timestamp": "2020-12-03T14:42:57,800Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "es-docker-cluster", "node.name": "es02", "message": "master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and this node must discover master-eligible nodes [es01, es02, es03] to bootstrap a cluster: have discovered [{es02}{YJiCiKnMSnuwqWrQGF0ojg}{OPwfpbcmQuicOGrm6aR-tg}{172.20.0.2}{172.20.0.2:9300}{dilmrt}{ml.machine_memory=16643383296, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}]; discovery will continue using [172.20.0.3:9300, 172.20.0.4:9300] from hosts providers and [{es02}{YJiCiKnMSnuwqWrQGF0ojg}{OPwfpbcmQuicOGrm6aR-tg}{172.20.0.2}{172.20.0.2:9300}{dilmrt}{ml.machine_memory=16643383296, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 0 in term 0" }

I am traying to use docker on centos 8 , the docker-compose file I'm using is this one:

5

Same as above: it's looking for multiple nodes but only discovering one:

The addresses look reasonable here, they're not just localhost, so in your case I'd guess it's a connectivity issue.

6

Thanks for your reply David,

...but now I am confused, I am running this on a single node via docker, iptables has no rules and selinux is Permisive, how can this be a connectivity issue ?

7

Sorry I've no idea, it says it's looking for the other nodes at those addresses, but it's not discovering them. Perhaps there's other log messages indicating why; you could try setting logger.org.elasticsearch.discovery: DEBUG, perhaps that'll show some useful exceptions happening.

8

Ready.

Disable firewalld, restart docker and done

9

You are using 'firewalld' inside a docker container? or on ECS instance?
Cause this service is not an enabled service in ECS enabled EC2 instance.
Something else might have happened. Cause the perview of the issue in docker or kubernetes is completely different than ECS.
If you found and resolved this issue in ECS platform (not Fargate) please let us know
Thanks in advance

10

Hi @s0umen, sorry I didn't realise this was your thread originally and it got sidetracked. As I mentioned above, I suggest setting logger.org.elasticsearch.discovery: DEBUG to get more visibility into the discovery process and any failures that are occurring there. If you need help understanding the resulting logs, please share them here.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.