Recently some of shards disappeared and can not be recovered.( 4 shard lost of 6 month logstash with 5 shard + 1replica)
I have changed replica from 1 to 0 due to storage shortage. but I don't know why this made some shard removed both primary and replica.
Because of this cluster's health is RED and cluster rolling restart doesn't help.
I have nothing can do so decided consider this to normal even there is some lost data.
how can I make cluster health to green in this situation?
So you have 5 nodes and 2800 shards. Means 560 shards per node.
Probably 280 days of data so far.
That's a lot IMO.
I'm unsure if you really need to have 5 shards per daily index. I would most likely reduce that in the logstash template to 1 shard, 1 replica. Also, if you don't use all indices, you can close them.
To answer to your question, if you don't really care about the incomplete indices, you can always delete them.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.