# New index immediately becomes red **URL:** \ **Category:** Elasticsearch\ **Created:** [November 19, 2013, 8:25pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474 "2013-11-19T20:25:09Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Andreas\_Echavez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_echavez/32/1959_2.png) [@Andreas\_Echavez](https://discuss.elastic.co/u/Andreas_Echavez)\ **Post date:** [November 19, 2013, 8:25pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/1 "2013-11-19T20:25:09Z")
Hi, I'm seeing some problems with a two-node cluster where all my shards have gone Yellow, and new shards are refusing to allocate. Both running 90.5 & Java 1.7.0\_25 I've tried almost everything I can think of to get my existing indexes passed initialization (routing API, reducing and increasing replicas, restarting and clearing out nodes) but no luck. Any ideas on how to troubleshoot this one? curl -XPUT '[http://localhost:9200/test\_index/](http://localhost:9200/test_index/)' > {"ok":true,"acknowledged":false} curl -s -XGET > '[http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | grep > test\_index -A15 > "test\_index" : { > "status" : "red", > "number\_of\_shards" : 5, > "number\_of\_replicas" : 1, > "active\_primary\_shards" : 0, > "active\_shards" : 0, > "relocating\_shards" : 0, > "initializing\_shards" : 3, > "unassigned\_shards" : 7 > }, Shards seem to be stuck in initializing... Node (os) stats [https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt](https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt) Cluster health [https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt](https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt) [https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json](https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vaidik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaidik/32/1134_2.png) [@vaidik](https://discuss.elastic.co/u/vaidik)\ **Post date:** [November 19, 2013, 8:30pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/2 "2013-11-19T20:30:21Z")
How many shards do you have on your node (i.e. no. of indices \* no. of shards \* no. of replicas) ? I was seeing a similar issues when I created a lot of indices with a high number of shards on a 2 node cluster. In that case, it is probably because of limited file descriptors. But that's my assumption and I am fairly new to this world. Vaidik Kapoor vaidikkapoor.info On 20 November 2013 01:55, Andreas Echavez [oceanplexian@gmail.com](mailto:oceanplexian@gmail.com) wrote: > Hi, > > I'm seeing some problems with a two-node cluster where all my shards have > gone Yellow, and new shards are refusing to allocate. > Both running 90.5 & Java 1.7.0\_25 > > I've tried almost everything I can think of to get my existing indexes > passed initialization (routing API, reducing and increasing replicas, > restarting and clearing out nodes) but no luck. Any ideas on how to > troubleshoot this one? > > curl -XPUT '[http://localhost:9200/test\_index/](http://localhost:9200/test_index/)' > > > {"ok":true,"acknowledged":false} > > curl -s -XGET ' > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | grep > > test\_index -A15 > > "test\_index" : { > > "status" : "red", > > "number\_of\_shards" : 5, > > "number\_of\_replicas" : 1, > > "active\_primary\_shards" : 0, > > "active\_shards" : 0, > > "relocating\_shards" : 0, > > "initializing\_shards" : 3, > > "unassigned\_shards" : 7 > > }, > > Shards seem to be stuck in initializing... > > Node (os) stats > > [https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt](https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt) > > Cluster health > > [https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt](https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt) > > [https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json](https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Kurt\_Hurtado](https://avatars.discourse-cdn.com/v4/letter/k/bbce88/32.png) [@Kurt\_Hurtado](https://discuss.elastic.co/u/Kurt_Hurtado)\ **Post date:** [November 19, 2013, 8:37pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/3 "2013-11-19T20:37:00Z")
Hi Andreas, You said you reduced and increase replicas... have you also experimented with number of shards? Best, Kurt On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote: > How many shards do you have on your node (i.e. no. of indices \* no. of > shards \* no. of replicas) ? I was seeing a similar issues when I created a > lot of indices with a high number of shards on a 2 node cluster. In that > case, it is probably because of limited file descriptors. But that's my > assumption and I am fairly new to this world. > > Vaidik Kapoor > vaidikkapoor.info > > On 20 November 2013 01:55, Andreas Echavez \<[oceanp...@gmail.com](mailto:oceanp...@gmail.com)\ > > > wrote: > > > Hi, > > > > I'm seeing some problems with a two-node cluster where all my shards have > > gone Yellow, and new shards are refusing to allocate. > > Both running 90.5 & Java 1.7.0\_25 > > > > I've tried almost everything I can think of to get my existing indexes > > passed initialization (routing API, reducing and increasing replicas, > > restarting and clearing out nodes) but no luck. Any ideas on how to > > troubleshoot this one? > > > > curl -XPUT '[http://localhost:9200/test\_index/](http://localhost:9200/test_index/)' > > > > > {"ok":true,"acknowledged":false} > > > > curl -s -XGET ' > > > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | grep > > > test\_index -A15 > > > "test\_index" : { > > > "status" : "red", > > > "number\_of\_shards" : 5, > > > "number\_of\_replicas" : 1, > > > "active\_primary\_shards" : 0, > > > "active\_shards" : 0, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 3, > > > "unassigned\_shards" : 7 > > > }, > > > > Shards seem to be stuck in initializing... > > > > Node (os) stats > > > > [https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt](https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt) > > > > Cluster health > > > > [https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt](https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt) > > > > [https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json](https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andreas\_Echavez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_echavez/32/1959_2.png) [@Andreas\_Echavez](https://discuss.elastic.co/u/Andreas_Echavez)\ **Post date:** [November 19, 2013, 8:38pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/4 "2013-11-19T20:38:16Z")
ATM it is set to 300k file descriptors which should cover it. I've got about ~220 shards across ~25 indexes or so. Thanks for the advice though, I did just double check it to be sure. [https://gist.github.com/oceanplexian/70f243b8482016174ce1/raw/b91dec063e9b3a548acf261487a0de29b6a7ce7b/gistfile1.txt](https://gist.github.com/oceanplexian/70f243b8482016174ce1/raw/b91dec063e9b3a548acf261487a0de29b6a7ce7b/gistfile1.txt) Thanks, Andreas On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote: > How many shards do you have on your node (i.e. no. of indices \* no. of > shards \* no. of replicas) ? I was seeing a similar issues when I created a > lot of indices with a high number of shards on a 2 node cluster. In that > case, it is probably because of limited file descriptors. But that's my > assumption and I am fairly new to this world. > > Vaidik Kapoor > vaidikkapoor.info > > On 20 November 2013 01:55, Andreas Echavez \<[oceanp...@gmail.com](mailto:oceanp...@gmail.com)\ > > > wrote: > > > Hi, > > > > I'm seeing some problems with a two-node cluster where all my shards have > > gone Yellow, and new shards are refusing to allocate. > > Both running 90.5 & Java 1.7.0\_25 > > > > I've tried almost everything I can think of to get my existing indexes > > passed initialization (routing API, reducing and increasing replicas, > > restarting and clearing out nodes) but no luck. Any ideas on how to > > troubleshoot this one? > > > > curl -XPUT '[http://localhost:9200/test\_index/](http://localhost:9200/test_index/)' > > > > > {"ok":true,"acknowledged":false} > > > > curl -s -XGET ' > > > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | grep > > > test\_index -A15 > > > "test\_index" : { > > > "status" : "red", > > > "number\_of\_shards" : 5, > > > "number\_of\_replicas" : 1, > > > "active\_primary\_shards" : 0, > > > "active\_shards" : 0, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 3, > > > "unassigned\_shards" : 7 > > > }, > > > > Shards seem to be stuck in initializing... > > > > Node (os) stats > > > > [https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt](https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt) > > > > Cluster health > > > > [https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt](https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt) > > > > [https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json](https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json) > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andreas\_Echavez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_echavez/32/1959_2.png) [@Andreas\_Echavez](https://discuss.elastic.co/u/Andreas_Echavez)\ **Post date:** [November 19, 2013, 8:45pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/5 "2013-11-19T20:45:00Z")
Trying with 1 shard and 0 replicas seems to be the same strange behavior curl -XPUT '[http://localhost:9200/test\_index4/](http://localhost:9200/test_index4/)' -d '{ > ``` > "settings" : { > "index" : { > "number_of_shards" : 1, > "number_of_replicas" : 0 > } > } > > ``` > > }' > {"ok":true,"acknowledged":false}root@es4:/data/elasticsearch# curl -s -XGET > '[http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | grep > test\_index4 -A10 > "test\_index4" : { > "status" : "red", > "number\_of\_shards" : 1, > "number\_of\_replicas" : 0, > "active\_primary\_shards" : 0, > "active\_shards" : 0, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 1 > }, Thanks, Andreas On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote: > Hi Andreas, > You said you reduced and increase replicas... have you also experimented > with number of shards? > Best, > Kurt > > On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote: > > > How many shards do you have on your node (i.e. no. of indices \* no. of > > shards \* no. of replicas) ? I was seeing a similar issues when I created a > > lot of indices with a high number of shards on a 2 node cluster. In that > > case, it is probably because of limited file descriptors. But that's my > > assumption and I am fairly new to this world. > > > > Vaidik Kapoor > > vaidikkapoor.info > > > > On 20 November 2013 01:55, Andreas Echavez [oceanp...@gmail.com](mailto:oceanp...@gmail.com) wrote: > > > > > Hi, > > > > > > I'm seeing some problems with a two-node cluster where all my shards > > > have gone Yellow, and new shards are refusing to allocate. > > > Both running 90.5 & Java 1.7.0\_25 > > > > > > I've tried almost everything I can think of to get my existing indexes > > > passed initialization (routing API, reducing and increasing replicas, > > > restarting and clearing out nodes) but no luck. Any ideas on how to > > > troubleshoot this one? > > > > > > curl -XPUT '[http://localhost:9200/test\_index/](http://localhost:9200/test_index/)' > > > > > > > {"ok":true,"acknowledged":false} > > > > > > curl -s -XGET ' > > > > > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | > > > > grep test\_index -A15 > > > > "test\_index" : { > > > > "status" : "red", > > > > "number\_of\_shards" : 5, > > > > "number\_of\_replicas" : 1, > > > > "active\_primary\_shards" : 0, > > > > "active\_shards" : 0, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 3, > > > > "unassigned\_shards" : 7 > > > > }, > > > > > > Shards seem to be stuck in initializing... > > > > > > Node (os) stats > > > > > > [https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt](https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt) > > > > > > Cluster health > > > > > > [https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt](https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt) > > > > > > [https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json](https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json) > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Kurt\_Hurtado](https://avatars.discourse-cdn.com/v4/letter/k/bbce88/32.png) [@Kurt\_Hurtado](https://discuss.elastic.co/u/Kurt_Hurtado)\ **Post date:** [November 19, 2013, 8:57pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/6 "2013-11-19T20:57:50Z")
So (assuming) you have time-based indices, do you close them once they are no longer being written to? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. On Tuesday, November 19, 2013 12:45:00 PM UTC-8, Andreas Echavez wrote: > Trying with 1 shard and 0 replicas seems to be the same strange behavior > > curl -XPUT '[http://localhost:9200/test\_index4/](http://localhost:9200/test_index4/)' -d '{ > > > ``` > > "settings" : { > > "index" : { > > "number_of_shards" : 1, > > "number_of_replicas" : 0 > > } > > } > > > > ``` > > > > }' > > {"ok":true,"acknowledged":false}root@es4:/data/elasticsearch# > > curl -s -XGET ' > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | grep > > test\_index4 -A10 > > "test\_index4" : { > > "status" : "red", > > "number\_of\_shards" : 1, > > "number\_of\_replicas" : 0, > > "active\_primary\_shards" : 0, > > "active\_shards" : 0, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 1 > > }, > > Thanks, > Andreas > > On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote: > > > Hi Andreas, > > You said you reduced and increase replicas... have you also experimented > > with number of shards? > > Best, > > Kurt > > > > On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote: > > > > > How many shards do you have on your node (i.e. no. of indices \* no. of > > > shards \* no. of replicas) ? I was seeing a similar issues when I created a > > > lot of indices with a high number of shards on a 2 node cluster. In that > > > case, it is probably because of limited file descriptors. But that's my > > > assumption and I am fairly new to this world. > > > > > > Vaidik Kapoor > > > vaidikkapoor.info > > > > > > On 20 November 2013 01:55, Andreas Echavez [oceanp...@gmail.com](mailto:oceanp...@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > I'm seeing some problems with a two-node cluster where all my shards > > > > have gone Yellow, and new shards are refusing to allocate. > > > > Both running 90.5 & Java 1.7.0\_25 > > > > > > > > I've tried almost everything I can think of to get my existing indexes > > > > passed initialization (routing API, reducing and increasing replicas, > > > > restarting and clearing out nodes) but no luck. Any ideas on how to > > > > troubleshoot this one? > > > > > > > > curl -XPUT '[http://localhost:9200/test\_index/](http://localhost:9200/test_index/)' > > > > > > > > > {"ok":true,"acknowledged":false} > > > > > > > > curl -s -XGET ' > > > > > > > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | > > > > > grep test\_index -A15 > > > > > "test\_index" : { > > > > > "status" : "red", > > > > > "number\_of\_shards" : 5, > > > > > "number\_of\_replicas" : 1, > > > > > "active\_primary\_shards" : 0, > > > > > "active\_shards" : 0, > > > > > "relocating\_shards" : 0, > > > > > "initializing\_shards" : 3, > > > > > "unassigned\_shards" : 7 > > > > > }, > > > > > > > > Shards seem to be stuck in initializing... > > > > > > > > Node (os) stats > > > > > > > > [https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt](https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt) > > > > > > > > Cluster health > > > > > > > > [https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt](https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt) > > > > > > > > [https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json](https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json) > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andreas\_Echavez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_echavez/32/1959_2.png) [@Andreas\_Echavez](https://discuss.elastic.co/u/Andreas_Echavez)\ **Post date:** [November 19, 2013, 10:11pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/7 "2013-11-19T22:11:58Z")
Unfortunately I don't have time based indices nor do we open/close them. Interestingly enough (for the sake of testing) it refuses to let me close an index because the primary isn't allocated {"error":"RemoteTransportException[[elasticsearch-prod-5][inet[/10.156.208.43:9300]][indices/close]]; > > nested: IndexPrimaryShardNotAllocatedException[[test\_index4] primary not > > allocated post api]; ","status":409} If anyone had any other ideas it'd be greatly appreciated. It really feels like a disk full or out of memory problem, but writing to existing indexes works fine, just not creating new ones or routing shards. Thanks, Andreas On Tuesday, November 19, 2013 12:57:50 PM UTC-8, Kurt Hurtado wrote: > So (assuming) you have time-based indices, do you close them once they are > no longer being written to? > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-open-close.html) > > On Tuesday, November 19, 2013 12:45:00 PM UTC-8, Andreas Echavez wrote: > > > Trying with 1 shard and 0 replicas seems to be the same strange behavior > > > > curl -XPUT '[http://localhost:9200/test\_index4/](http://localhost:9200/test_index4/)' -d '{ > > > > > ``` > > > "settings" : { > > > "index" : { > > > "number_of_shards" : 1, > > > "number_of_replicas" : 0 > > > } > > > } > > > > > > ``` > > > > > > }' > > > {"ok":true,"acknowledged":false}root@es4:/data/elasticsearch# > > > > curl -s -XGET ' > > > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | grep > > > test\_index4 -A10 > > > "test\_index4" : { > > > "status" : "red", > > > "number\_of\_shards" : 1, > > > "number\_of\_replicas" : 0, > > > "active\_primary\_shards" : 0, > > > "active\_shards" : 0, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 0, > > > "unassigned\_shards" : 1 > > > }, > > > > Thanks, > > Andreas > > > > On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote: > > > > > Hi Andreas, > > > You said you reduced and increase replicas... have you also experimented > > > with number of shards? > > > Best, > > > Kurt > > > > > > On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote: > > > > > > > How many shards do you have on your node (i.e. no. of indices \* no. of > > > > shards \* no. of replicas) ? I was seeing a similar issues when I created a > > > > lot of indices with a high number of shards on a 2 node cluster. In that > > > > case, it is probably because of limited file descriptors. But that's my > > > > assumption and I am fairly new to this world. > > > > > > > > Vaidik Kapoor > > > > vaidikkapoor.info > > > > > > > > On 20 November 2013 01:55, Andreas Echavez [oceanp...@gmail.com](mailto:oceanp...@gmail.com) wrote: > > > > > > > > > Hi, > > > > > > > > > > I'm seeing some problems with a two-node cluster where all my shards > > > > > have gone Yellow, and new shards are refusing to allocate. > > > > > Both running 90.5 & Java 1.7.0\_25 > > > > > > > > > > I've tried almost everything I can think of to get my existing indexes > > > > > passed initialization (routing API, reducing and increasing replicas, > > > > > restarting and clearing out nodes) but no luck. Any ideas on how to > > > > > troubleshoot this one? > > > > > > > > > > curl -XPUT '[http://localhost:9200/test\_index/](http://localhost:9200/test_index/)' > > > > > > > > > > > {"ok":true,"acknowledged":false} > > > > > > > > > > curl -s -XGET ' > > > > > > > > > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | > > > > > > grep test\_index -A15 > > > > > > "test\_index" : { > > > > > > "status" : "red", > > > > > > "number\_of\_shards" : 5, > > > > > > "number\_of\_replicas" : 1, > > > > > > "active\_primary\_shards" : 0, > > > > > > "active\_shards" : 0, > > > > > > "relocating\_shards" : 0, > > > > > > "initializing\_shards" : 3, > > > > > > "unassigned\_shards" : 7 > > > > > > }, > > > > > > > > > > Shards seem to be stuck in initializing... > > > > > > > > > > Node (os) stats > > > > > > > > > > [https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt](https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt) > > > > > > > > > > Cluster health > > > > > > > > > > [https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt](https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt) > > > > > > > > > > [https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json](https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json) > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andreas\_Echavez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_echavez/32/1959_2.png) [@Andreas\_Echavez](https://discuss.elastic.co/u/Andreas_Echavez)\ **Post date:** [November 21, 2013, 5:32pm UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/8 "2013-11-21T17:32:13Z")
So I figured this out for anyone encountering a similar issue: Looked through the logs and found this error being displayed when ES tried to create some indexes: Caused by: org.elasticsearch.env.FailedToResolveConfigException: Failed to > resolve config path [/etc/elasticsearch/synonyms/location\_synonyms.txt], > tried file path [/etc/elasticsearch/synonyms/location\_synonyms.txt], path > file [/etc/elasticsearch/etc/elasticsearch/synonyms/location\_synonyms.txt], > and classpath Looks like the term synonyms file was blocking shards from being allocated. Even though the files were in the listed locations (note they are relative to the conf dir) with all permissions verified, I ended up having to dump them in the classpath for ES to finally pick them up. That was the trick. Everything is back to normal and recovering. Thanks! Andreas On Tuesday, November 19, 2013 2:11:58 PM UTC-8, Andreas Echavez wrote: > Unfortunately I don't have time based indices nor do we open/close them. > Interestingly enough (for the sake of testing) it refuses to let me close > an index because the primary isn't allocated > > {"error":"RemoteTransportException[[elasticsearch-prod-5][inet[/10.156.208.43:9300]][indices/close]]; > > > > nested: IndexPrimaryShardNotAllocatedException[[test\_index4] primary not > > > allocated post api]; ","status":409} > > If anyone had any other ideas it'd be greatly appreciated. It really feels > like a disk full or out of memory problem, but writing to existing indexes > works fine, just not creating new ones or routing shards. > > Thanks, > Andreas > > On Tuesday, November 19, 2013 12:57:50 PM UTC-8, Kurt Hurtado wrote: > > > So (assuming) you have time-based indices, do you close them once they > > are no longer being written to? > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-open-close.html) > > > > On Tuesday, November 19, 2013 12:45:00 PM UTC-8, Andreas Echavez wrote: > > > > > Trying with 1 shard and 0 replicas seems to be the same strange behavior > > > > > > curl -XPUT '[http://localhost:9200/test\_index4/](http://localhost:9200/test_index4/)' -d '{ > > > > > > > ``` > > > > "settings" : { > > > > "index" : { > > > > "number_of_shards" : 1, > > > > "number_of_replicas" : 0 > > > > } > > > > } > > > > > > > > ``` > > > > > > > > }' > > > > {"ok":true,"acknowledged":false}root@es4:/data/elasticsearch# > > > > > > curl -s -XGET ' > > > > > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | > > > > grep test\_index4 -A10 > > > > "test\_index4" : { > > > > "status" : "red", > > > > "number\_of\_shards" : 1, > > > > "number\_of\_replicas" : 0, > > > > "active\_primary\_shards" : 0, > > > > "active\_shards" : 0, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 0, > > > > "unassigned\_shards" : 1 > > > > }, > > > > > > Thanks, > > > Andreas > > > > > > On Tuesday, November 19, 2013 12:37:00 PM UTC-8, Kurt Hurtado wrote: > > > > > > > Hi Andreas, > > > > You said you reduced and increase replicas... have you also > > > > experimented with number of shards? > > > > Best, > > > > Kurt > > > > > > > > On Tuesday, November 19, 2013 12:30:21 PM UTC-8, Vaidik Kapoor wrote: > > > > > > > > > How many shards do you have on your node (i.e. no. of indices \* no. of > > > > > shards \* no. of replicas) ? I was seeing a similar issues when I created a > > > > > lot of indices with a high number of shards on a 2 node cluster. In that > > > > > case, it is probably because of limited file descriptors. But that's my > > > > > assumption and I am fairly new to this world. > > > > > > > > > > Vaidik Kapoor > > > > > vaidikkapoor.info > > > > > > > > > > On 20 November 2013 01:55, Andreas Echavez [oceanp...@gmail.com](mailto:oceanp...@gmail.com)wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > I'm seeing some problems with a two-node cluster where all my shards > > > > > > have gone Yellow, and new shards are refusing to allocate. > > > > > > Both running 90.5 & Java 1.7.0\_25 > > > > > > > > > > > > I've tried almost everything I can think of to get my existing > > > > > > indexes passed initialization (routing API, reducing and increasing > > > > > > replicas, restarting and clearing out nodes) but no luck. Any ideas on how > > > > > > to troubleshoot this one? > > > > > > > > > > > > curl -XPUT '[http://localhost:9200/test\_index/](http://localhost:9200/test_index/)' > > > > > > > > > > > > > {"ok":true,"acknowledged":false} > > > > > > > > > > > > curl -s -XGET ' > > > > > > > > > > > > > [http://localhost:9200/\_cluster/health?pretty=true&level=indices](http://localhost:9200/_cluster/health?pretty=true&level=indices)' | > > > > > > > grep test\_index -A15 > > > > > > > "test\_index" : { > > > > > > > "status" : "red", > > > > > > > "number\_of\_shards" : 5, > > > > > > > "number\_of\_replicas" : 1, > > > > > > > "active\_primary\_shards" : 0, > > > > > > > "active\_shards" : 0, > > > > > > > "relocating\_shards" : 0, > > > > > > > "initializing\_shards" : 3, > > > > > > > "unassigned\_shards" : 7 > > > > > > > }, > > > > > > > > > > > > Shards seem to be stuck in initializing... > > > > > > > > > > > > Node (os) stats > > > > > > > > > > > > [https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt](https://gist.github.com/oceanplexian/7551916/raw/5a950e7a20d2a6d6b73a0086d0753300cc6dcba8/gistfile1.txt) > > > > > > > > > > > > Cluster health > > > > > > > > > > > > [https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt](https://gist.github.com/oceanplexian/03a88abc3f14e30b145a/raw/5f7a2367e566b33d46b19fce7df2062403a8d97b/gistfile1.txt) > > > > > > > > > > > > [https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json](https://gist.github.com/oceanplexian/7551756/raw/4b0cf68930c3a34e84bd6eae7ce4886400d4b1a2/gistfile1.json) > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:05am UTC](https://discuss.elastic.co/t/new-index-immediately-becomes-red/14474/9 "2017-07-06T02:05:37Z")