# Elasticsearch red status **URL:** \ **Category:** Elasticsearch\ **Created:** [January 9, 2013, 9:25am UTC](https://discuss.elastic.co/t/elasticsearch-red-status/10276 "2013-01-09T09:25:24Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [January 9, 2013, 9:25am UTC](https://discuss.elastic.co/t/elasticsearch-red-status/10276/1 "2013-01-09T09:25:24Z")
Hi, We have a cluster with 17 nodes, about 30 index and each index 50 shards using elasticsearch 0.19.9 we have a big problem with getting the ES status yellow. Some times it never get yellow even after lots of restarting and waiting. I can see in head plugin that just one primary shard doesn't get allocated. Is there any solution to solve it? or is it possible to find which node cause the problem to restart only that node? Of course I read in this forum that restarting the cluster could solve the problem(maybe more than 30 times!). But in our case is not possible to restart the cluster so many times. Thanks, Vahid -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 9, 2013, 9:33am UTC](https://discuss.elastic.co/t/elasticsearch-red-status/10276/2 "2013-01-09T09:33:48Z")
Hello Vahid, That unallocated shard causes yellow or red status? I mean, is there a replica allocated for that shard? If yes, that's automatically promoted to primary. And you should have the quick solution of reducing the number of replicas by 1, and increasing it back again. You can find out where the shard belongs to, but you'd have to look in all the nodes, somewhere like this: $DATA\_DIR/$CLUSTER\_NAME/nodes/0/indices/$INDEX\_NAME/$SHARD\_NUMBER The node that has the directory but doesn't have the shard allocated to it contains the troublesome shard. ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Wed, Jan 9, 2013 at 11:25 AM, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote: > Hi, > > We have a cluster with 17 nodes, about 30 index and each index 50 shards > using elasticsearch 0.19.9 > we have a big problem with getting the ES status yellow. Some times it > never get yellow even after lots of restarting and waiting. I can see in > head plugin that just one primary shard doesn't get allocated. > Is there any solution to solve it? or is it possible to find which node > cause the problem to restart only that node? > Of course I read in this forum that restarting the cluster could solve the > problem(maybe more than 30 times!). But in our case is not possible to > restart the cluster so many times. > > Thanks, > Vahid > > -- -- ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [January 9, 2013, 10:37am UTC](https://discuss.elastic.co/t/elasticsearch-red-status/10276/3 "2013-01-09T10:37:28Z")
Hi Radu, Thank you for your reply, The cluster status is RED and never get yellow or green. There is one replica configured, and for one shards both primary and replica don't get allocated so the status is red and also there is no shard path(as you mentioned) with such shard number. In my case, there must be a folder with such path "$DATA\_DIR/$CLUSTER\_NAME/nodes/0/indices/$INDEX\_NAME/8", but there is no folder. (8 is the not allocated shards number) Best regards, Vahid On Wednesday, January 9, 2013 10:33:48 AM UTC+1, Radu Gheorghe wrote: > Hello Vahid, > > That unallocated shard causes yellow or red status? I mean, is there a > replica allocated for that shard? If yes, that's automatically promoted to > primary. And you should have the quick solution of reducing the number of > replicas by 1, and increasing it back again. > > You can find out where the shard belongs to, but you'd have to look in all > the nodes, somewhere like this: > > $DATA\_DIR/$CLUSTER\_NAME/nodes/0/indices/$INDEX\_NAME/$SHARD\_NUMBER > > The node that has the directory but doesn't have the shard allocated to it > contains the troublesome shard. > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Wed, Jan 9, 2013 at 11:25 AM, Vahid \<[vhas...@gmail.com](mailto:vhas...@gmail.com) \\>wrote: > > > Hi, > > > > We have a cluster with 17 nodes, about 30 index and each index 50 shards > > using elasticsearch 0.19.9 > > we have a big problem with getting the ES status yellow. Some times it > > never get yellow even after lots of restarting and waiting. I can see in > > head plugin that just one primary shard doesn't get allocated. > > Is there any solution to solve it? or is it possible to find which node > > cause the problem to restart only that node? > > Of course I read in this forum that restarting the cluster could solve > > the problem(maybe more than 30 times!). But in our case is not possible to > > restart the cluster so many times. > > > > Thanks, > > Vahid > > > > -- -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 9, 2013, 11:28am UTC](https://discuss.elastic.co/t/elasticsearch-red-status/10276/4 "2013-01-09T11:28:56Z")
Hi Vahid, OK, so if you don't have the shard contents anywhere you'll probably have to reindex to get your data back. Or restore from backup. If you want to know why the shard disappeared (so you can prevent this from happening again), you have to look for cluesin the time before it disappeared. Did you do a full cluster restart? Is there something relevant in the logs? ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Wed, Jan 9, 2013 at 12:37 PM, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote: > Hi Radu, > Thank you for your reply, > > The cluster status is RED and never get yellow or green. > There is one replica configured, and for one shards both primary and > replica don't get allocated so the status is red and also there is no shard > path(as you mentioned) with such shard number. > In my case, there must be a folder with such path > "$DATA\_DIR/$CLUSTER\_NAME/nodes/0/indices/$INDEX\_NAME/8", but there is no > folder. (8 is the not allocated shards number) > > Best regards, > Vahid > > On Wednesday, January 9, 2013 10:33:48 AM UTC+1, Radu Gheorghe wrote: > > > Hello Vahid, > > > > That unallocated shard causes yellow or red status? I mean, is there a > > replica allocated for that shard? If yes, that's automatically promoted to > > primary. And you should have the quick solution of reducing the number of > > replicas by 1, and increasing it back again. > > > > You can find out where the shard belongs to, but you'd have to look in > > all the nodes, somewhere like this: > > > > $DATA\_DIR/$CLUSTER\_NAME/nodes/\*\*0/indices/$INDEX\_NAME/$SHARD\_\*\*NUMBER > > > > The node that has the directory but doesn't have the shard allocated to > > it contains the troublesome shard. > > > > ## Best regards, Radu > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > On Wed, Jan 9, 2013 at 11:25 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > > Hi, > > > > > > We have a cluster with 17 nodes, about 30 index and each index 50 shards > > > using elasticsearch 0.19.9 > > > we have a big problem with getting the ES status yellow. Some times it > > > never get yellow even after lots of restarting and waiting. I can see in > > > head plugin that just one primary shard doesn't get allocated. > > > Is there any solution to solve it? or is it possible to find which node > > > cause the problem to restart only that node? > > > Of course I read in this forum that restarting the cluster could solve > > > the problem(maybe more than 30 times!). But in our case is not possible to > > > restart the cluster so many times. > > > > > > Thanks, > > > Vahid > > > > > > -- > > > > -- -- ---
**Author:** ![Vahid](https://avatars.discourse-cdn.com/v4/letter/v/96bed5/32.png) [@Vahid](https://discuss.elastic.co/u/Vahid)\ **Post date:** [January 9, 2013, 1:04pm UTC](https://discuss.elastic.co/t/elasticsearch-red-status/10276/5 "2013-01-09T13:04:00Z")
Hi Radu, Yes, I've just restarted whole the cluster, the only log I could see is the some things like this: "org.elasticsearch.index.IndexShardMissingException: [INDEX\_NAME][37] missing" that I was observing such exception during the cluster start up before, but there was no problem. Previously even by such exceptions, after a while ES was getting yellow and working. Anyway thank you, now I know that I have some data lost. Best regards, Vahid On Wednesday, January 9, 2013 12:28:56 PM UTC+1, Radu Gheorghe wrote: > Hi Vahid, > > OK, so if you don't have the shard contents anywhere you'll probably have > to reindex to get your data back. Or restore from backup. > > If you want to know why the shard disappeared (so you can prevent this > from happening again), you have to look for cluesin the time before it > disappeared. Did you do a full cluster restart? Is there something relevant > in the logs? > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Wed, Jan 9, 2013 at 12:37 PM, Vahid \<[vhas...@gmail.com](mailto:vhas...@gmail.com) \\>wrote: > > > Hi Radu, > > Thank you for your reply, > > > > The cluster status is RED and never get yellow or green. > > There is one replica configured, and for one shards both primary and > > replica don't get allocated so the status is red and also there is no shard > > path(as you mentioned) with such shard number. > > In my case, there must be a folder with such path > > "$DATA\_DIR/$CLUSTER\_NAME/nodes/0/indices/$INDEX\_NAME/8", but there is no > > folder. (8 is the not allocated shards number) > > > > Best regards, > > Vahid > > > > On Wednesday, January 9, 2013 10:33:48 AM UTC+1, Radu Gheorghe wrote: > > > > > Hello Vahid, > > > > > > That unallocated shard causes yellow or red status? I mean, is there a > > > replica allocated for that shard? If yes, that's automatically promoted to > > > primary. And you should have the quick solution of reducing the number of > > > replicas by 1, and increasing it back again. > > > > > > You can find out where the shard belongs to, but you'd have to look in > > > all the nodes, somewhere like this: > > > > > > $DATA\_DIR/$CLUSTER\_NAME/nodes/\*\*0/indices/$INDEX\_NAME/$SHARD\_\*\*NUMBER > > > > > > The node that has the directory but doesn't have the shard allocated to > > > it contains the troublesome shard. > > > > > > ## Best regards, Radu > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > On Wed, Jan 9, 2013 at 11:25 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > We have a cluster with 17 nodes, about 30 index and each index 50 > > > > shards using elasticsearch 0.19.9 > > > > we have a big problem with getting the ES status yellow. Some times it > > > > never get yellow even after lots of restarting and waiting. I can see in > > > > head plugin that just one primary shard doesn't get allocated. > > > > Is there any solution to solve it? or is it possible to find which node > > > > cause the problem to restart only that node? > > > > Of course I read in this forum that restarting the cluster could solve > > > > the problem(maybe more than 30 times!). But in our case is not possible to > > > > restart the cluster so many times. > > > > > > > > Thanks, > > > > Vahid > > > > > > > > -- > > > > > > -- -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 9, 2013, 3:57pm UTC](https://discuss.elastic.co/t/elasticsearch-red-status/10276/6 "2013-01-09T15:57:40Z")
You're welcome 🙂 Just one quick note: make sure you have appropriate recovery[0] and minimum\_master\_nodes[1] settings. Otherwise, restarting your cluster might lead to dangling indices or split brain. [0] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/modules/gateway/) [1] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/modules/discovery/zen.html) ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Wed, Jan 9, 2013 at 3:04 PM, Vahid [vhasani57@gmail.com](mailto:vhasani57@gmail.com) wrote: > Hi Radu, > > Yes, I've just restarted whole the cluster, the only log I could see is > the some things like this: > "org.elasticsearch.index.IndexShardMissingException: [INDEX\_NAME][37] > missing" > that I was observing such exception during the cluster start up before, > but there was no problem. Previously even by such exceptions, after a while > ES was getting yellow and working. > Anyway thank you, now I know that I have some data lost. > > Best regards, > Vahid > > On Wednesday, January 9, 2013 12:28:56 PM UTC+1, Radu Gheorghe wrote: > > > Hi Vahid, > > > > OK, so if you don't have the shard contents anywhere you'll probably have > > to reindex to get your data back. Or restore from backup. > > > > If you want to know why the shard disappeared (so you can prevent this > > from happening again), you have to look for cluesin the time before it > > disappeared. Did you do a full cluster restart? Is there something relevant > > in the logs? > > > > ## Best regards, Radu > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > On Wed, Jan 9, 2013 at 12:37 PM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > > Hi Radu, > > > Thank you for your reply, > > > > > > The cluster status is RED and never get yellow or green. > > > There is one replica configured, and for one shards both primary and > > > replica don't get allocated so the status is red and also there is no shard > > > path(as you mentioned) with such shard number. > > > In my case, there must be a folder with such path > > > "$DATA\_DIR/$CLUSTER\_NAME/ **nodes/0/indices/$INDEX\_NAME/8"** , but there > > > is no folder. (8 is the not allocated shards number) > > > > > > Best regards, > > > Vahid > > > > > > On Wednesday, January 9, 2013 10:33:48 AM UTC+1, Radu Gheorghe wrote: > > > > > > > Hello Vahid, > > > > > > > > That unallocated shard causes yellow or red status? I mean, is there a > > > > replica allocated for that shard? If yes, that's automatically promoted to > > > > primary. And you should have the quick solution of reducing the number of > > > > replicas by 1, and increasing it back again. > > > > > > > > You can find out where the shard belongs to, but you'd have to look in > > > > all the nodes, somewhere like this: > > > > > > > > $DATA\_DIR/$CLUSTER\_NAME/nodes/\*\*\*\*0/indices/$INDEX\_NAME/$SHARD\_ **N** > > > > UMBER > > > > > > > > The node that has the directory but doesn't have the shard allocated to > > > > it contains the troublesome shard. > > > > > > > > ## Best regards, Radu > > > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > > > On Wed, Jan 9, 2013 at 11:25 AM, Vahid [vhas...@gmail.com](mailto:vhas...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > We have a cluster with 17 nodes, about 30 index and each index 50 > > > > > shards using elasticsearch 0.19.9 > > > > > we have a big problem with getting the ES status yellow. Some times it > > > > > never get yellow even after lots of restarting and waiting. I can see in > > > > > head plugin that just one primary shard doesn't get allocated. > > > > > Is there any solution to solve it? or is it possible to find which > > > > > node cause the problem to restart only that node? > > > > > Of course I read in this forum that restarting the cluster could solve > > > > > the problem(maybe more than 30 times!). But in our case is not possible to > > > > > restart the cluster so many times. > > > > > > > > > > Thanks, > > > > > Vahid > > > > > > > > > > -- > > > > > > > > -- > > > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:57am UTC](https://discuss.elastic.co/t/elasticsearch-red-status/10276/7 "2017-07-06T02:57:12Z")