# How did we get into a yellow state and how do we get out of there? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 24, 2010, 2:37pm UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597 "2010-11-24T14:37:27Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![tkroehling](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tkroehling/32/3302_2.png) [@tkroehling](https://discuss.elastic.co/u/tkroehling)\ **Post date:** [November 24, 2010, 2:37pm UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597/1 "2010-11-24T14:37:27Z")
Hi, we have a cluster of 3 search servers on Amazon EC2s and are feeding it extensively with documents. We have a configuration with 3 replicas and 5 shards. At startup everything was fine. But now when I look at the health of the cluster I see a state yellow and some unassigned shards: { "cluster\_name" : "warp", "status" : "yellow", "timed\_out" : false, "number\_of\_nodes" : 5, "number\_of\_data\_nodes" : 3, "active\_primary\_shards" : 10, "active\_shards" : 30, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 10 } I cannot find any log messages about any problems that may have occurred or any actions taken in order to get back to a cluster state green. Log level is "info". Is there anything I need to do to get back to state green? Or is there hope that the system heals itself? Is there a way to get more information about what the server is doing, like trying to heal itself? Well, I guess I could switch the log level to debug. But maybe something more specific. Or is there maybe something wrong with my configuration? Do I need to configure more shards? We currently have almost a million documents (which is not really a lot), taking up about 10GB, but we have only imported about 10% of the expected data. Any help appreciated. Thanks, Tom ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [November 24, 2010, 2:51pm UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597/2 "2010-11-24T14:51:24Z")
Hi > we have a cluster of 3 search servers on Amazon EC2s and are feeding > it extensively with documents. > We have a configuration with 3 replicas and 5 shards. Note, 3 replicas = 4 nodes (1 master + 3 replicas). You are only running 3 data nodes, so 1 replica is unassigned. Green = all shards are assigned Yellow = all master shards are live Red = all master shards are not live So yellow is the normal situation in your setup. If you decrease the number of replicas to 2, then your cluster will become green. [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/indices/update\_settings/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indices/update_settings/) clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 24, 2010, 2:51pm UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597/3 "2010-11-24T14:51:48Z")
Hey, if you specify 3 replicas, this means you will have basically need to have 4 data nodes to get to green state. This is for the simple reason that a shard and its replicas are not allocated on the same node, and the total number of shard (1) and its replicas (3) is 4. -shay.baon On Wed, Nov 24, 2010 at 4:37 PM, tkroehling [thomas.kroehling@coremedia.com](mailto:thomas.kroehling@coremedia.com)wrote: > Hi, > we have a cluster of 3 search servers on Amazon EC2s and are feeding > it extensively with documents. > We have a configuration with 3 replicas and 5 shards. > At startup everything was fine. But now when I look at the health of > the cluster I see a state yellow and some unassigned shards: > { > "cluster\_name" : "warp", > "status" : "yellow", > "timed\_out" : false, > "number\_of\_nodes" : 5, > "number\_of\_data\_nodes" : 3, > "active\_primary\_shards" : 10, > "active\_shards" : 30, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 10 > } > I cannot find any log messages about any problems that may have > occurred or any actions taken in order to get back to a cluster state > green. Log level is "info". > Is there anything I need to do to get back to state green? Or is there > hope that the system heals itself? Is there a way to get more > information about what the server is doing, like trying to heal > itself? Well, I guess I could switch the log level to debug. But maybe > something more specific. > Or is there maybe something wrong with my configuration? Do I need to > configure more shards? > We currently have almost a million documents (which is not really a > lot), taking up about 10GB, but we have only imported about 10% of the > expected data. > Any help appreciated. > Thanks, > Tom ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [November 24, 2010, 2:52pm UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597/4 "2010-11-24T14:52:22Z")
bad english > Red = all master shards are not live Red = not all master shards are live clint ---
**Author:** ![tkroehling](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tkroehling/32/3302_2.png) [@tkroehling](https://discuss.elastic.co/u/tkroehling)\ **Post date:** [November 24, 2010, 2:58pm UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597/5 "2010-11-24T14:58:27Z")
That explains everything. I thought the total number of replicas would include the master. I will change my configuration. 2 replicas will be sufficient in my setup. Thanks a lot. Tom ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 24, 2010, 3:06pm UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597/6 "2010-11-24T15:06:59Z")
You can change that at runtime (the number of replicas), check this API: [http://www.elasticsearch.com/docs/elasticsearch/rest\_api/admin/indices/update\_settings/](http://www.elasticsearch.com/docs/elasticsearch/rest_api/admin/indices/update_settings/) On Wed, Nov 24, 2010 at 4:58 PM, tkroehling [thomas.kroehling@coremedia.com](mailto:thomas.kroehling@coremedia.com)wrote: > That explains everything. > I thought the total number of replicas would include the master. > I will change my configuration. 2 replicas will be sufficient in my > setup. > Thanks a lot. > Tom ---
**Author:** ![tkroehling](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tkroehling/32/3302_2.png) [@tkroehling](https://discuss.elastic.co/u/tkroehling)\ **Post date:** [November 24, 2010, 3:43pm UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597/7 "2010-11-24T15:43:53Z")
That's exactly what I did to reconfigure it and it worked well. Everything is up and running. And the cluster also recovers now without any problems from manual server crashes in our tests. Thanks again, Tom ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:16am UTC](https://discuss.elastic.co/t/how-did-we-get-into-a-yellow-state-and-how-do-we-get-out-of-there/3597/8 "2017-07-06T04:16:07Z")