Stop elasticsearch cluster safely

Elastic Stack Elasticsearch
1

Hello, there.

If anyone has faced a similar problem, I would really appreciate your advice.

We stop/start Elasticsearch cluster in develop envrionment everyday to reduce costs.

But, sometimes, we find that some index`status is red after starting cluster.
Primary and secondary shards is lost.

We think there is a problem with the way we stop/start and are looking for a better way.

Our environment confgiuretion is here.

================================
Elasticserch version is 7.8.1.
Our cluster have 3 nodes and those are default role (It's a master, it's a datanode, it`s other.).
We run Elasticsearch in docker container and these container is running in AWS EC2.
To stop cluster, we stop AWS EC2 Instances simply.

2

Can you share your docker configuration file? Also cluster status is recovered automatically or it remains in the RED status?

3

Thanks for you reply!

Our docker configuration is bellow.We don't use AWS ECS.
※To use on ec2, we install dicovery-ec2 plugin.
※In EFS(like a NFS), we set dictionary (don't set shards.)

docker run \
--log-driver=awslogs \
--log-opt awslogs-region=${region} \
--log-opt awslogs-group=${LogGroup} \
--log-opt awslogs-stream=${instanceid} \
--ulimit nofile=102400:102400 \
-d --restart=always \
--net host -e "ES_JAVA_OPTS=-Xms${EsJavaXmsValue} -Xmx${EsJavaXmxValue} -Dlog4j2.formatMsgNoLookups=true" \
-e cluster.name=${name} \
-e TAKE_FILE_OWNERSHIP=true \
-e cluster.initial_master_nodes=$nodenames \
-e discovery.seed_hosts=$nodenames \
-e discovery.ec2.any_group=true \
-e discovery.ec2.host_type=private_ip \
-e discovery.seed_providers=ec2 \
-e cloud.node.auto_attributes=true \
-e cluster.routing.allocation.awareness.attributes=aws_availability_zone \
-e discovery.ec2.groups=${name}\ 
discovery.ec2.availability_zones=${AvailabilityZone1},${AvailabilityZone2} \
-v /usr/share/elasticsearch/data:/usr/share/elasticsearch/data \
-v /mnt/efs:/mnt/efs

We use official image by elastic.(docker.elastic.co/elasticsearch/elasticsearch:7.8.1)
Our docker file install plugins and place dictionary file simply.

Also cluster status is recovered automatically or it remains in the RED status?

Yes.
A day goes by, and the Index never recovers...
We re-index to recover it.

4

Do you loose all your data ie all your indices and shards when you stop and start or only few of them?

5

Only few of them.
We hava indexes of some types, but, there is no regularity in the index type of loss...

6

What type of instance are you using? What type of storage?

7

@Christian_Dahlqvist
Hi, thanks for replying.

What type of instance are you using? What type of storage?

We use t3.medium for develop environment.
EBS type is gp3.
IOPS is 3000, throughput is 125.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.