How can we create an Elasticsearch cluster with AWS ECS and Docker

[chamiduz (Chamidu)]

Hi,

I'm trying to create an ES cluster with AWS ECS and Docker. here is what i have already done.

1. created aws infrastructure

2. ECS cluster

3. Network load balancer

4. target groups for 9200,9300 ports and attached them to the container

5. discovery.seed_hosts can be retrieved using the load balancer because i could not find any method to update the #discovery.seed_hosts whenever a new node is created

   i tried network LB with a ES cluster hosted in two EC2s and it worked
   however when i try to use ES Docker container it fails to join the cluster

2020-02-20T09:52:57,445][INFO ][o.e.c.c.JoinHelper ] [data-2] failed to join {data-1}{YwBabAZSQOy832Hzgxg_Lg}{Y_s5CmoEQ5-YtYqBO736DQ}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}{dim}{ml.machine_memory=8095113216, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={data-2}{HOJG_exiRiqfjZjd77ZH_Q}{wJr1_FscQ8KtXgF1v6jpVg}{192.168.0.20}{192.168.0.20:9300}{dilm}{ml.machine_memory=8272052224, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=58, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={data-2}{HOJG_exiRiqfjZjd77ZH_Q}{wJr1_FscQ8KtXgF1v6jpVg}{192.168.0.20}{192.168.0.20:9300}{dilm}{ml.machine_memory=8272052224, xpack.installed=true, ml.max_open_jobs=20}, targetNode={data-1}{YwBabAZSQOy832Hzgxg_Lg}{Y_s5CmoEQ5-YtYqBO736DQ}{xx.xx.xx.xx}{xx.xx.xx.xx:9300}{dim}{ml.machine_memory=8095113216, ml.max_open_jobs=20, xpack.installed=true}}]}

xx.xx.xx.xx is the loadbancer ip

[DavidTurner]

You have only shared the first line of a much longer message indicating that this node failed to join a cluster, and the rest of the message tells us why.

@chamiduz

discovery.seed_hosts can be retrieved using the load balancer because i could not find any method to update the #discovery.seed_hosts whenever a new node is created

Either set discovery.seed_hosts: dns.name.example.org where dns.name.example.org resolves via DNS to the addresses of all the master-eligible nodes, or else use the file-based seed hosts provider. DNS-based discovery should play nicely with ECS's own service discovery mechanism.

@chamiduz

xx.xx.xx.xx is the loadbancer ip

This sounds wrong since this is also the address that data-1 claims is its own. It does not work to hide nodes from each other using a load balancer: they must be able to connect to each other directly.