First time cluster installation- cluster status red

Hello,

I am trying to setup my first ES cluster with 3 nodes, all master. During the setup of the cluster I was rebooting the ES service on the nodes which seems to have corrupted some of the indexes in the nodes(I believe). I then removed the cluster configuration from elasticsearch.yml hoping this would make all the nodes all single nodes, not within the cluster anymore.
When checking the indexes, I see a number of them are in state red.

curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
red open logstash-2017.05.20 yLfLmOpJQ4S4J8Ha4rucOQ 5 1 8365 0 8.1mb 8.1mb

I left the default setting for replication to 1, which I wrongly assumed that it wont replicate the data between the nodes.
My questions are:
1)can I recover from index in state red? Or do I need to delete it and start from scratch?
2)will all my indexes replicate in cluster by default?
3)Some of my indexes seem to have not been affected. If I restart the service of ES after I changed the configuration of the cluster in order to remove it from the cluster, is it possible they will get corrupted further?
If so, can I do a backup of index on the fly without having to configure a backup path, which would make me restart the service?

Also I see the following when executing command:
curl -XPOST "http://localhost:920GET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

logstash-2017.09.01 4 p UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 2 p UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 3 p UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 1 p UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 1 r UNASSIGNED CLUSTER_RECOVERED

They all show unassigned, cluster_recovered.

At this stage took out this node out of my 3 node cluster and want it to be isolated on its own.
Is it possible to fix or does it look like my data is corrupted?

It seems like you might be having quite small shards, and possibly quite a few of them, which can be inefficient and cause problems. I would recommend reading the following blog post about shards and sharding.

What is the output of the cluster stats API?

Thanks for the link, really useful. Setup my cluster from scratch and it works!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.