First time cluster installation- cluster status red


#1

Hello,

I am trying to setup my first ES cluster with 3 nodes, all master. During the setup of the cluster I was rebooting the ES service on the nodes which seems to have corrupted some of the indexes in the nodes(I believe). I then removed the cluster configuration from elasticsearch.yml hoping this would make all the nodes all single nodes, not within the cluster anymore.
When checking the indexes, I see a number of them are in state red.

curl -XGET 'localhost:9200/_cat/indices?v&pretty'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
red open logstash-2017.05.20 yLfLmOpJQ4S4J8Ha4rucOQ 5 1 8365 0 8.1mb 8.1mb

I left the default setting for replication to 1, which I wrongly assumed that it wont replicate the data between the nodes.
My questions are:
1)can I recover from index in state red? Or do I need to delete it and start from scratch?
2)will all my indexes replicate in cluster by default?
3)Some of my indexes seem to have not been affected. If I restart the service of ES after I changed the configuration of the cluster in order to remove it from the cluster, is it possible they will get corrupted further?
If so, can I do a backup of index on the fly without having to configure a backup path, which would make me restart the service?


#2

Also I see the following when executing command:
curl -XPOST "http://localhost:920GET localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason| grep UNASSIGNED

logstash-2017.09.01 4 p UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 4 r UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 2 p UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 2 r UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 3 p UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 3 r UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 1 p UNASSIGNED CLUSTER_RECOVERED
logstash-2017.09.01 1 r UNASSIGNED CLUSTER_RECOVERED

They all show unassigned, cluster_recovered.

At this stage took out this node out of my 3 node cluster and want it to be isolated on its own.
Is it possible to fix or does it look like my data is corrupted?


(Christian Dahlqvist) #3

It seems like you might be having quite small shards, and possibly quite a few of them, which can be inefficient and cause problems. I would recommend reading the following blog post about shards and sharding.

What is the output of the cluster stats API?


#4

Thanks for the link, really useful. Setup my cluster from scratch and it works!


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.