Elasticsearch on AWS ECS Fargate

Elastic Stack Elasticsearch
#1

Hi. I apologize if this is a duplicate of another thread, but so far I did not see a similar case.

I am trying to deploy ES on ECS with default image from docker.elastic.co/elasticsearch/elasticsearch:7.11.1 but it is failing with error

ERROR: [3] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65535]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[3]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log

My target is to run multiple nodes connected to EFS and load balanced by a Load Balancer, but if single node works that will also be fine.

Your help would be highly appreciated.

#2

Welcome!

Please read Bootstrap Checks | Elasticsearch Reference [7.11] | Elastic

#3

Thanks David, To include the changes, I had to bake my own ES docker Image. And while i need to have discovery in place for other containers to communicate with each other and appoint Master Node, I created elasticsearch.yml file and Dockerfile and build/deployed to ECS Fargate, but it is failing with errors as shown below:

{"type": "server", "timestamp": "2021-02-17T15:28:00,564Z", "level": "ERROR", "component": "o.e.d.e.Ec2DiscoveryPlugin", "cluster.name": "elasticsearch", "node.name": "ip-10-0-1-195.ap-south-1.compute.internal", "message": "failed to get metadata for [placement/availability-zone]",

{"type": "server", "timestamp": "2021-02-17T15:28:11,585Z", "level": "ERROR", "component": "o.e.b.ElasticsearchUncaughtExceptionHandler", "cluster.name": "elasticsearch", "node.name": "ip-10-0-1-195.ap-south-1.compute.internal", "message": "uncaught exception in thread [main]",

"stacktrace ["org.elasticsearch.bootstrap.StartupException: BindTransportException[Failed to resolve publish address]; nested: IOException[IOException caught when fetching InetAddress from [http://169.254.169.254/latest/meta-data/local-ipv4]]; nested: SocketException[Invalid argument];",

elasticsearch.yml

cluster.name: "elasticsearch"
path.data: /var/data/elasticsearch
path.logs: /var/logs/elasticsearch
bootstrap.memory_lock: false
network.host: 0.0.0.0
network.publish_host: _ec2:privateIp_
transport.publish_host: _ec2:privateIp_
discovery.zen.hosts_provider: ec2
discovery.ec2.tag.ElasticSearch: prod
discovery.ec2.endpoint: ec2.${REGION}.amazonaws.com
discovery.type: single-node
s3.client.default.endpoint: s3.${REGION}.amazonaws.com
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
xpack.security.enabled: false

Dockerfile

FROM docker.elastic.co/elasticsearch/elasticsearch:7.11.0
ENV REGION us-east-1
ADD elasticsearch.yml /usr/share/elasticsearch/config/
USER root
RUN chown elasticsearch:elasticsearch config/elasticsearch.yml
USER elasticsearch
WORKDIR /usr/share/elasticsearch
RUN bin/elasticsearch-plugin install -b discovery-ec2 && bin/elasticsearch-plugin install -b repository-s3 && sed -e '/^-Xm/s/^/#/g' -i /usr/share/elasticsearch/config/jvm.options
#4 
IOException caught when fetching InetAddress from [http://169.254.169.254/latest...

You're using _ec2:privateIp_ as your publish host, which means "look up my address from the EC2 metadata service", but it appears that the EC2 metadata service is not accessible within this container. I think that makes sense, you're running this in ECS Fargate not directly in EC2, so the discovery-ec2 plugin isn't going to work for you.

#5

In essence this discovery-ec2 plugin should work for ecs fargate as well, but if it does not, Does Elasticsearch offer a workaround for this ?

I am being forced to test Single Node container without discovery and see the outcome.

#6

I don't think it should, no, ECS Fargate has its own DNS-based service discovery mechanism that you should use instead.

#7

You are right David. ECS Fargate uses Service Discovery and i will need to consider to use it. For now I will launch Single Node.

1 Like