Kibana cannot use elasticsearch normally - Cluster Red

Elastic Stack Elasticsearch
1

es

kib
kib2165×384 78.4 KB

Configuration environment: es-7.17.8 kibana-7.17.8 filebeat-7.17.8
(Elasticsearch is a single node, and the data mount point is the efs file system of AWS)

Problem Description:

I started to find that kibanan was in use. When searching for the load, there was a prompt that some shards could not be loaded. Check the index.
It was found that some indexes were in red, and the indexes of some machines in the last two days were not collected into es normally.
Start restarting es, and the machines that have not collected the index are still unable to collect normally. Then we check and find that the number of primary and secondary shards of the index is 1.
Then configure the number of index shards as primary 1 and secondary 0. The yellow index becomes green, and the red index remains red.
The status of elasticsearch is still red, and there is always the number of shards of unassigned_shards.
Restarting elasticsearch several times still fails to convert the number of shards in unassigned_shards to the number of shards in the active_shards state.
Reinstalled kibanan many times (operation process: stop kibanan, delete kibanan related indexes in elasticsearch, reinstall kibanan)

The final status of elasticsearch and kibanan is as shown in the figure

2

According to the official best practices for AWS the use of EFS is not recommended or supported for node storage. This could very well explain the issues you are seeing. If some indices are red it is possible that they are corrupted and that the data is lost. Check the Elasticsearch logs for further details.

As you only have 1 node you may need to restore the red shards from a snapshot or simply delete them in order to get the cluster back to green.

It also looks like you have more shards on the node than is allowed by default, so you have apparently overridden that setting. Note that these limits are in place for a good reason. I would recommend you read this blog post.

3

image
image1216×150 29.5 KB

Hello, after stopping filebeat collection, delete other red indexes that have little impact. Currently, these indexes are in red status. The current problem is that the index mode in kibbana cannot be created and an error is reported.

4

You may need to delete and recreate, ideally by restoring from a snapshot, the red indices.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.