# How to resolve elasticsearch status red
**URL:** https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401
**Category:** Elasticsearch
**Created:** [July 12, 2012, 6:51pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401 "2012-07-12T18:51:01Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@Yuhan](https://discuss.elastic.co/u/Yuhan)
#### Post date: [July 12, 2012, 6:51pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/1 "2012-07-12T18:51:01Z")
Hi all,
I got "No Active Record" exception on some index but works for other.
The health was indicating status=red, and there are unassigned\_shards.
{
"cluster\_name" : "elasticsearch",
"status" : "red",
"timed\_out" : false,
"number\_of\_nodes" : 1,
"number\_of\_data\_nodes" : 1,
"active\_primary\_shards" : 54,
"active\_shards" : 54,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 66
}
what should I do to correct this situration? could someone give some
recommended reading?
Thank you.
Yuhan
---
### Author:  [@drewr](https://discuss.elastic.co/u/drewr)
#### Post date: [July 12, 2012, 8:45pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/2 "2012-07-12T20:45:13Z")
Yuhan Zhang wrote:
> I got "No Active Record" exception on some index but works for other.
> The health was indicating status=red, and there are unassigned\_shards.
>
> {
> "cluster\_name" : "elasticsearch",
> "status" : "red",
> "timed\_out" : false,
> "number\_of\_nodes" : 1,
> "number\_of\_data\_nodes" : 1,
> "active\_primary\_shards" : 54,
> "active\_shards" : 54,
> "relocating\_shards" : 0,
> "initializing\_shards" : 0,
> "unassigned\_shards" : 66
> }
>
> what should I do to correct this situration? could someone give some
> recommended reading?
You should check the master log (this node, since there's only one)
for exceptions or other information about why they are not assigning.
-Drew
---
### Author:  [@browe](https://discuss.elastic.co/u/browe)
#### Post date: [August 5, 2012, 5:26pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/3 "2012-08-05T17:26:05Z")
My situation is similar. I have not been able to resolve this or find any
solution online yet..
My cluster health:
{
"cluster\_name" : "elasticsearch",
"status" : "red",
"timed\_out" : false,
"number\_of\_nodes" : 1,
"number\_of\_data\_nodes" : 1,
"active\_primary\_shards" : 22,
"active\_shards" : 22,
"relocating\_shards" : 0,
"initializing\_shards" : 0,
"unassigned\_shards" : 58
}
I can do searches, but if I try an XPUT it fails. This did work at one
time.
curl -XPUT '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{
"user" : "kimchy",
"post\_date" : "2009-11-15T14:12:12",
"message" : "trying out Elastic Search"
}'
{"error":"UnavailableShardsException[[twitter][2] [2] shardIt, [0] active :
Timeout waiting for [1m], request: index {[twitter][tweet][1], source[{\n
"user" : "kimchy",\n "post\_date" : "2009-11-15T14:12:12",\n
"message" : "trying out Elastic Search"\n}]}]","status":503
What also appears odd is that my failed PUTS don't show up in the log. But
when I shutdown and startup the server, those activities do show up in the
log. ( I deleted the old log to start over and try everything again)
On Thursday, July 12, 2012 2:51:01 PM UTC-4, Yuhan wrote:
> Hi all,
>
> I got "No Active Record" exception on some index but works for other.
> The health was indicating status=red, and there are unassigned\_shards.
>
> {
> "cluster\_name" : "elasticsearch",
> "status" : "red",
> "timed\_out" : false,
> "number\_of\_nodes" : 1,
> "number\_of\_data\_nodes" : 1,
> "active\_primary\_shards" : 54,
> "active\_shards" : 54,
> "relocating\_shards" : 0,
> "initializing\_shards" : 0,
> "unassigned\_shards" : 66
> }
>
> what should I do to correct this situration? could someone give some
> recommended reading?
>
> Thank you.
>
> Yuhan
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [August 5, 2012, 5:58pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/4 "2012-08-05T17:58:07Z")
My opinion is that you created some indexes and some of them with no replica.
You started more than one node in your LAN. Then, you shutdown one node.
ES can not give you a green or yellow health as some of your documents (index with no replica) can not be seen by ES.
Is that what happened?
David
--
Le 5 août 2012 à 19:26, browe [browe@perceivant.com](mailto:browe@perceivant.com) a écrit :
> My situation is similar. I have not been able to resolve this or find any solution online yet..
> My cluster health:
> {
> "cluster\_name" : "elasticsearch",
> "status" : "red",
> "timed\_out" : false,
> "number\_of\_nodes" : 1,
> "number\_of\_data\_nodes" : 1,
> "active\_primary\_shards" : 22,
> "active\_shards" : 22,
> "relocating\_shards" : 0,
> "initializing\_shards" : 0,
> "unassigned\_shards" : 58
> }
> I can do searches, but if I try an XPUT it fails. This did work at one time.
>
> curl -XPUT '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{
> "user" : "kimchy",
> "post\_date" : "2009-11-15T14:12:12",
> "message" : "trying out Elastic Search"
> }'
> {"error":"UnavailableShardsException[[twitter][2] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[twitter][tweet][1], source[{\n "user" : "kimchy",\n "post\_date" : "2009-11-15T14:12:12",\n "message" : "trying out Elastic Search"\n}]}]","status":503
>
> What also appears odd is that my failed PUTS don't show up in the log. But when I shutdown and startup the server, those activities do show up in the log. ( I deleted the old log to start over and try everything again)
>
> On Thursday, July 12, 2012 2:51:01 PM UTC-4, Yuhan wrote:
> Hi all,
>
> I got "No Active Record" exception on some index but works for other.
> The health was indicating status=red, and there are unassigned\_shards.
>
> {
> "cluster\_name" : "elasticsearch",
> "status" : "red",
> "timed\_out" : false,
> "number\_of\_nodes" : 1,
> "number\_of\_data\_nodes" : 1,
> "active\_primary\_shards" : 54,
> "active\_shards" : 54,
> "relocating\_shards" : 0,
> "initializing\_shards" : 0,
> "unassigned\_shards" : 66
> }
>
> what should I do to correct this situration? could someone give some recommended reading?
>
> Thank you.
>
> Yuhan
---
### Author:  [@browe](https://discuss.elastic.co/u/browe)
#### Post date: [August 5, 2012, 6:48pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/5 "2012-08-05T18:48:47Z")
I'm not sure because I really just started ES on 1 server with the default
configuration and started trying to put test data into it. I haven't
tried to add replica or change the configuration. I have created some
indexes. And some that I would like to delete, but I can't PUT any
commands to even delete data. I don't think there has ever been more than
one node, but I really don't even know how to start more than one node. I
only restart the ES server using the init commands as a service. I have 58
unassigned shards, but I don't know how to correct them. I don't really
need any of the data either, I just need to be Green again, so I can start
over, but I would like to figure out what happened so I can not do it again
when I do need the data.
On Sunday, August 5, 2012 1:58:07 PM UTC-4, David Pilato wrote:
> My opinion is that you created some indexes and some of them with no
> replica.
> You started more than one node in your LAN. Then, you shutdown one node.
>
> ES can not give you a green or yellow health as some of your documents
> (index with no replica) can not be seen by ES.
>
> Is that what happened?
>
> David
>
> --
>
> Le 5 août 2012 à 19:26, browe [browe@perceivant.com](mailto:browe@perceivant.com) a écrit :
>
> My situation is similar. I have not been able to resolve this or find any
> solution online yet..
> My cluster health:
> {
> "cluster\_name" : "elasticsearch",
> "status" : "red",
> "timed\_out" : false,
> "number\_of\_nodes" : 1,
> "number\_of\_data\_nodes" : 1,
> "active\_primary\_shards" : 22,
> "active\_shards" : 22,
> "relocating\_shards" : 0,
> "initializing\_shards" : 0,
> "unassigned\_shards" : 58
> }
> I can do searches, but if I try an XPUT it fails. This did work at one
> time.
>
> curl -XPUT '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{
> "user" : "kimchy",
> "post\_date" : "2009-11-15T14:12:12",
> "message" : "trying out Elastic Search"
> }'
> {"error":"UnavailableShardsException[[twitter][2] [2] shardIt, [0] active
> : Timeout waiting for [1m], request: index {[twitter][tweet][1], source[{\n
> "user" : "kimchy",\n "post\_date" : "2009-11-15T14:12:12",\n
> "message" : "trying out Elastic Search"\n}]}]","status":503
>
> What also appears odd is that my failed PUTS don't show up in the log.
> But when I shutdown and startup the server, those activities do show up in
> the log. ( I deleted the old log to start over and try everything again)
>
> On Thursday, July 12, 2012 2:51:01 PM UTC-4, Yuhan wrote:
>
> > Hi all,
> >
> > I got "No Active Record" exception on some index but works for other.
> > The health was indicating status=red, and there are unassigned\_shards.
> >
> > {
> > "cluster\_name" : "elasticsearch",
> > "status" : "red",
> > "timed\_out" : false,
> > "number\_of\_nodes" : 1,
> > "number\_of\_data\_nodes" : 1,
> > "active\_primary\_shards" : 54,
> > "active\_shards" : 54,
> > "relocating\_shards" : 0,
> > "initializing\_shards" : 0,
> > "unassigned\_shards" : 66
> > }
> >
> > what should I do to correct this situration? could someone give some
> > recommended reading?
> >
> > Thank you.
> >
> > Yuhan
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [August 5, 2012, 6:55pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/6 "2012-08-05T18:55:57Z")
So, stop your node and delete data dir.
You will restart a clean node.
Are you sure that nobody else start a node on your LAN?
You should at least change the cluster name if you have coworkers.
David
--
Le 5 août 2012 à 20:48, browe [browe@perceivant.com](mailto:browe@perceivant.com) a écrit :
> I'm not sure because I really just started ES on 1 server with the default configuration and started trying to put test data into it. I haven't tried to add replica or change the configuration. I have created some indexes. And some that I would like to delete, but I can't PUT any commands to even delete data. I don't think there has ever been more than one node, but I really don't even know how to start more than one node. I only restart the ES server using the init commands as a service. I have 58 unassigned shards, but I don't know how to correct them. I don't really need any of the data either, I just need to be Green again, so I can start over, but I would like to figure out what happened so I can not do it again when I do need the data.
>
> On Sunday, August 5, 2012 1:58:07 PM UTC-4, David Pilato wrote:
> My opinion is that you created some indexes and some of them with no replica.
> You started more than one node in your LAN. Then, you shutdown one node.
>
> ES can not give you a green or yellow health as some of your documents (index with no replica) can not be seen by ES.
>
> Is that what happened?
>
> David
>
> --
>
> Le 5 août 2012 à 19:26, browe [browe@perceivant.com](mailto:browe@perceivant.com) a écrit :
>
> > My situation is similar. I have not been able to resolve this or find any solution online yet..
> > My cluster health:
> > {
> > "cluster\_name" : "elasticsearch",
> > "status" : "red",
> > "timed\_out" : false,
> > "number\_of\_nodes" : 1,
> > "number\_of\_data\_nodes" : 1,
> > "active\_primary\_shards" : 22,
> > "active\_shards" : 22,
> > "relocating\_shards" : 0,
> > "initializing\_shards" : 0,
> > "unassigned\_shards" : 58
> > }
> > I can do searches, but if I try an XPUT it fails. This did work at one time.
> >
> > curl -XPUT '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{
> > "user" : "kimchy",
> > "post\_date" : "2009-11-15T14:12:12",
> > "message" : "trying out Elastic Search"
> > }'
> > {"error":"UnavailableShardsException[[twitter][2] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[twitter][tweet][1], source[{\n "user" : "kimchy",\n "post\_date" : "2009-11-15T14:12:12",\n "message" : "trying out Elastic Search"\n}]}]","status":503
> >
> > What also appears odd is that my failed PUTS don't show up in the log. But when I shutdown and startup the server, those activities do show up in the log. ( I deleted the old log to start over and try everything again)
> >
> > On Thursday, July 12, 2012 2:51:01 PM UTC-4, Yuhan wrote:
> > Hi all,
> >
> > I got "No Active Record" exception on some index but works for other.
> > The health was indicating status=red, and there are unassigned\_shards.
> >
> > {
> > "cluster\_name" : "elasticsearch",
> > "status" : "red",
> > "timed\_out" : false,
> > "number\_of\_nodes" : 1,
> > "number\_of\_data\_nodes" : 1,
> > "active\_primary\_shards" : 54,
> > "active\_shards" : 54,
> > "relocating\_shards" : 0,
> > "initializing\_shards" : 0,
> > "unassigned\_shards" : 66
> > }
> >
> > what should I do to correct this situration? could someone give some recommended reading?
> >
> > Thank you.
> >
> > Yuhan
---
### Author:  [@browe](https://discuss.elastic.co/u/browe)
#### Post date: [August 5, 2012, 7:39pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/7 "2012-08-05T19:39:31Z")
yeah.. deleting the data made it green.. So I can start over. but I
certainly couldn't do that if i needed the data in the future when that
happens.. A little scary to keep moving forward with this solution if that
is the way to fix issues... Thank you for the help. I don't think I really
learned anything..
On Sunday, August 5, 2012 2:55:57 PM UTC-4, David Pilato wrote:
> So, stop your node and delete data dir.
> You will restart a clean node.
>
> Are you sure that nobody else start a node on your LAN?
>
> You should at least change the cluster name if you have coworkers.
>
> David
>
> --
>
> Le 5 août 2012 à 20:48, browe a écrit :
>
> I'm not sure because I really just started ES on 1 server with the default
> configuration and started trying to put test data into it. I haven't
> tried to add replica or change the configuration. I have created some
> indexes. And some that I would like to delete, but I can't PUT any
> commands to even delete data. I don't think there has ever been more than
> one node, but I really don't even know how to start more than one node. I
> only restart the ES server using the init commands as a service. I have 58
> unassigned shards, but I don't know how to correct them. I don't really
> need any of the data either, I just need to be Green again, so I can start
> over, but I would like to figure out what happened so I can not do it again
> when I do need the data.
>
> On Sunday, August 5, 2012 1:58:07 PM UTC-4, David Pilato wrote:
>
> > My opinion is that you created some indexes and some of them with no
> > replica.
> > You started more than one node in your LAN. Then, you shutdown one node.
> >
> > ES can not give you a green or yellow health as some of your documents
> > (index with no replica) can not be seen by ES.
> >
> > Is that what happened?
> >
> > David
> >
> > --
> >
> > Le 5 août 2012 à 19:26, browe a écrit :
> >
> > My situation is similar. I have not been able to resolve this or find
> > any solution online yet..
> > My cluster health:
> > {
> > "cluster\_name" : "elasticsearch",
> > "status" : "red",
> > "timed\_out" : false,
> > "number\_of\_nodes" : 1,
> > "number\_of\_data\_nodes" : 1,
> > "active\_primary\_shards" : 22,
> > "active\_shards" : 22,
> > "relocating\_shards" : 0,
> > "initializing\_shards" : 0,
> > "unassigned\_shards" : 58
> > }
> > I can do searches, but if I try an XPUT it fails. This did work at one
> > time.
> >
> > curl -XPUT '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{
> > "user" : "kimchy",
> > "post\_date" : "2009-11-15T14:12:12",
> > "message" : "trying out Elastic Search"
> > }'
> > {"error":"UnavailableShardsException[[twitter][2] [2] shardIt, [0] active
> > : Timeout waiting for [1m], request: index {[twitter][tweet][1], source[{\n
> > "user" : "kimchy",\n "post\_date" : "2009-11-15T14:12:12",\n
> > "message" : "trying out Elastic Search"\n}]}]","status":503
> >
> > What also appears odd is that my failed PUTS don't show up in the log.
> > But when I shutdown and startup the server, those activities do show up in
> > the log. ( I deleted the old log to start over and try everything again)
> >
> > On Thursday, July 12, 2012 2:51:01 PM UTC-4, Yuhan wrote:
> >
> > > Hi all,
> > >
> > > I got "No Active Record" exception on some index but works for other.
> > > The health was indicating status=red, and there are unassigned\_shards.
> > >
> > > {
> > > "cluster\_name" : "elasticsearch",
> > > "status" : "red",
> > > "timed\_out" : false,
> > > "number\_of\_nodes" : 1,
> > > "number\_of\_data\_nodes" : 1,
> > > "active\_primary\_shards" : 54,
> > > "active\_shards" : 54,
> > > "relocating\_shards" : 0,
> > > "initializing\_shards" : 0,
> > > "unassigned\_shards" : 66
> > > }
> > >
> > > what should I do to correct this situration? could someone give some
> > > recommended reading?
> > >
> > > Thank you.
> > >
> > > Yuhan
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [August 5, 2012, 8:06pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/8 "2012-08-05T20:06:35Z")
Let me say that I never went into such issue since I started with ES in production one year ago.
--
Le 5 août 2012 à 21:39, browe [browe@perceivant.com](mailto:browe@perceivant.com) a écrit :
> yeah.. deleting the data made it green.. So I can start over. but I certainly couldn't do that if i needed the data in the future when that happens.. A little scary to keep moving forward with this solution if that is the way to fix issues... Thank you for the help. I don't think I really learned anything..
>
> On Sunday, August 5, 2012 2:55:57 PM UTC-4, David Pilato wrote:
> So, stop your node and delete data dir.
> You will restart a clean node.
>
> Are you sure that nobody else start a node on your LAN?
>
> You should at least change the cluster name if you have coworkers.
>
> David
>
> --
>
> Le 5 août 2012 à 20:48, browe a écrit :
>
> > I'm not sure because I really just started ES on 1 server with the default configuration and started trying to put test data into it. I haven't tried to add replica or change the configuration. I have created some indexes. And some that I would like to delete, but I can't PUT any commands to even delete data. I don't think there has ever been more than one node, but I really don't even know how to start more than one node. I only restart the ES server using the init commands as a service. I have 58 unassigned shards, but I don't know how to correct them. I don't really need any of the data either, I just need to be Green again, so I can start over, but I would like to figure out what happened so I can not do it again when I do need the data.
> >
> > On Sunday, August 5, 2012 1:58:07 PM UTC-4, David Pilato wrote:
> > My opinion is that you created some indexes and some of them with no replica.
> > You started more than one node in your LAN. Then, you shutdown one node.
> >
> > ES can not give you a green or yellow health as some of your documents (index with no replica) can not be seen by ES.
> >
> > Is that what happened?
> >
> > David
> >
> > --
> >
> > Le 5 août 2012 à 19:26, browe a écrit :
> >
> > > My situation is similar. I have not been able to resolve this or find any solution online yet..
> > > My cluster health:
> > > {
> > > "cluster\_name" : "elasticsearch",
> > > "status" : "red",
> > > "timed\_out" : false,
> > > "number\_of\_nodes" : 1,
> > > "number\_of\_data\_nodes" : 1,
> > > "active\_primary\_shards" : 22,
> > > "active\_shards" : 22,
> > > "relocating\_shards" : 0,
> > > "initializing\_shards" : 0,
> > > "unassigned\_shards" : 58
> > > }
> > > I can do searches, but if I try an XPUT it fails. This did work at one time.
> > >
> > > curl -XPUT '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{
> > > "user" : "kimchy",
> > > "post\_date" : "2009-11-15T14:12:12",
> > > "message" : "trying out Elastic Search"
> > > }'
> > > {"error":"UnavailableShardsException[[twitter][2] [2] shardIt, [0] active : Timeout waiting for [1m], request: index {[twitter][tweet][1], source[{\n "user" : "kimchy",\n "post\_date" : "2009-11-15T14:12:12",\n "message" : "trying out Elastic Search"\n}]}]","status":503
> > >
> > > What also appears odd is that my failed PUTS don't show up in the log. But when I shutdown and startup the server, those activities do show up in the log. ( I deleted the old log to start over and try everything again)
> > >
> > > On Thursday, July 12, 2012 2:51:01 PM UTC-4, Yuhan wrote:
> > > Hi all,
> > >
> > > I got "No Active Record" exception on some index but works for other.
> > > The health was indicating status=red, and there are unassigned\_shards.
> > >
> > > {
> > > "cluster\_name" : "elasticsearch",
> > > "status" : "red",
> > > "timed\_out" : false,
> > > "number\_of\_nodes" : 1,
> > > "number\_of\_data\_nodes" : 1,
> > > "active\_primary\_shards" : 54,
> > > "active\_shards" : 54,
> > > "relocating\_shards" : 0,
> > > "initializing\_shards" : 0,
> > > "unassigned\_shards" : 66
> > > }
> > >
> > > what should I do to correct this situration? could someone give some recommended reading?
> > >
> > > Thank you.
> > >
> > > Yuhan
---
### Author:  [@Eran\_Kutner\_2](https://discuss.elastic.co/u/Eran_Kutner_2)
#### Post date: [August 8, 2012, 7:03am UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/9 "2012-08-08T07:03:14Z")
David,
The fact you didn't encounter a certain problem doesn't mean it's not there.
ES definitely has a problem with not loading all the shards after a
cluster/server restart. Restarting enough times in the right order usually
solves the issue. I've already reported it multiple times. Here is the most
recent one:
[https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/from:eran/elasticsearch/jwlyJQ7gg4s/0v-0e0hv7PoJ[1-25]](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/from:eran/elasticsearch/jwlyJQ7gg4s/0v-0e0hv7PoJ%5B1-25%5D)
By the way, it took probably 30 restarts to get the above cluster to load
all the shards.
-eran
On Sunday, August 5, 2012 11:06:35 PM UTC+3, David Pilato wrote:
> Let me say that I never went into such issue since I started with ES in
> production one year ago.
>
> --
>
> Le 5 août 2012 à 21:39, browe \<[br...@perceivant.com](mailto:br...@perceivant.com) \\> a
> écrit :
>
> yeah.. deleting the data made it green.. So I can start over. but I
> certainly couldn't do that if i needed the data in the future when that
> happens.. A little scary to keep moving forward with this solution if that
> is the way to fix issues... Thank you for the help. I don't think I really
> learned anything..
>
> On Sunday, August 5, 2012 2:55:57 PM UTC-4, David Pilato wrote:
>
> > So, stop your node and delete data dir.
> > You will restart a clean node.
> >
> > Are you sure that nobody else start a node on your LAN?
> >
> > You should at least change the cluster name if you have coworkers.
> >
> > David
> >
> > --
> >
> > Le 5 août 2012 à 20:48, browe a écrit :
> >
> > I'm not sure because I really just started ES on 1 server with the
> > default configuration and started trying to put test data into it. I
> > haven't tried to add replica or change the configuration. I have created
> > some indexes. And some that I would like to delete, but I can't PUT any
> > commands to even delete data. I don't think there has ever been more than
> > one node, but I really don't even know how to start more than one node. I
> > only restart the ES server using the init commands as a service. I have 58
> > unassigned shards, but I don't know how to correct them. I don't really
> > need any of the data either, I just need to be Green again, so I can start
> > over, but I would like to figure out what happened so I can not do it again
> > when I do need the data.
> >
> > On Sunday, August 5, 2012 1:58:07 PM UTC-4, David Pilato wrote:
> >
> > > My opinion is that you created some indexes and some of them with no
> > > replica.
> > > You started more than one node in your LAN. Then, you shutdown one node.
> > >
> > > ES can not give you a green or yellow health as some of your documents
> > > (index with no replica) can not be seen by ES.
> > >
> > > Is that what happened?
> > >
> > > David
> > >
> > > --
> > >
> > > Le 5 août 2012 à 19:26, browe a écrit :
> > >
> > > My situation is similar. I have not been able to resolve this or find
> > > any solution online yet..
> > > My cluster health:
> > > {
> > > "cluster\_name" : "elasticsearch",
> > > "status" : "red",
> > > "timed\_out" : false,
> > > "number\_of\_nodes" : 1,
> > > "number\_of\_data\_nodes" : 1,
> > > "active\_primary\_shards" : 22,
> > > "active\_shards" : 22,
> > > "relocating\_shards" : 0,
> > > "initializing\_shards" : 0,
> > > "unassigned\_shards" : 58
> > > }
> > > I can do searches, but if I try an XPUT it fails. This did work at one
> > > time.
> > >
> > > curl -XPUT '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{
> > > "user" : "kimchy",
> > > "post\_date" : "2009-11-15T14:12:12",
> > > "message" : "trying out Elastic Search"
> > > }'
> > > {"error":"UnavailableShardsException[[twitter][2] [2] shardIt, [0]
> > > active : Timeout waiting for [1m], request: index {[twitter][tweet][1],
> > > source[{\n "user" : "kimchy",\n "post\_date" :
> > > "2009-11-15T14:12:12",\n "message" : "trying out Elastic
> > > Search"\n}]}]","status":503
> > >
> > > What also appears odd is that my failed PUTS don't show up in the log.
> > > But when I shutdown and startup the server, those activities do show up in
> > > the log. ( I deleted the old log to start over and try everything again)
> > >
> > > On Thursday, July 12, 2012 2:51:01 PM UTC-4, Yuhan wrote:
> > >
> > > > Hi all,
> > > >
> > > > I got "No Active Record" exception on some index but works for other.
> > > > The health was indicating status=red, and there are unassigned\_shards.
> > > >
> > > > {
> > > > "cluster\_name" : "elasticsearch",
> > > > "status" : "red",
> > > > "timed\_out" : false,
> > > > "number\_of\_nodes" : 1,
> > > > "number\_of\_data\_nodes" : 1,
> > > > "active\_primary\_shards" : 54,
> > > > "active\_shards" : 54,
> > > > "relocating\_shards" : 0,
> > > > "initializing\_shards" : 0,
> > > > "unassigned\_shards" : 66
> > > > }
> > > >
> > > > what should I do to correct this situration? could someone give some
> > > > recommended reading?
> > > >
> > > > Thank you.
> > > >
> > > > Yuhan
---
### Author:  [@ywtsang](https://discuss.elastic.co/u/ywtsang)
#### Post date: [August 8, 2012, 12:20pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/10 "2012-08-08T12:20:03Z")
Hi eran,
I cannot visit your link and it redirects to the forum index page. Can
you check the link? As I want to read your reported issue.
And for the "right order required for cluster restart", I think I met
a similar one before:
[https://groups.google.com/d/msg/elasticsearch/JNh39-Ccrjo/UZYDgRint00J](https://groups.google.com/d/msg/elasticsearch/JNh39-Ccrjo/UZYDgRint00J)
for my situation, I also just try to restart each node one by one and
hope the cluster can recover by itself magically.
Thanks,
Wing
On Wed, Aug 8, 2012 at 3:03 PM, Eran [eran@gigya-inc.com](mailto:eran@gigya-inc.com) wrote:
> David,
> The fact you didn't encounter a certain problem doesn't mean it's not there.
> ES definitely has a problem with not loading all the shards after a
> cluster/server restart. Restarting enough times in the right order usually
> solves the issue. I've already reported it multiple times. Here is the most
> recent one:
> [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/from:eran/elasticsearch/jwlyJQ7gg4s/0v-0e0hv7PoJ%5B1-25%5D)
> By the way, it took probably 30 restarts to get the above cluster to load
> all the shards.
>
> -eran
>
> On Sunday, August 5, 2012 11:06:35 PM UTC+3, David Pilato wrote:
>
> > Let me say that I never went into such issue since I started with ES in
> > production one year ago.
> >
> > --
> >
> > Le 5 août 2012 à 21:39, browe [br...@perceivant.com](mailto:br...@perceivant.com) a écrit :
> >
> > yeah.. deleting the data made it green.. So I can start over. but I
> > certainly couldn't do that if i needed the data in the future when that
> > happens.. A little scary to keep moving forward with this solution if that
> > is the way to fix issues... Thank you for the help. I don't think I really
> > learned anything..
> >
> > On Sunday, August 5, 2012 2:55:57 PM UTC-4, David Pilato wrote:
> >
> > > So, stop your node and delete data dir.
> > > You will restart a clean node.
> > >
> > > Are you sure that nobody else start a node on your LAN?
> > >
> > > You should at least change the cluster name if you have coworkers.
> > >
> > > David
> > >
> > > --
> > >
> > > Le 5 août 2012 à 20:48, browe a écrit :
> > >
> > > I'm not sure because I really just started ES on 1 server with the
> > > default configuration and started trying to put test data into it. I
> > > haven't tried to add replica or change the configuration. I have created
> > > some indexes. And some that I would like to delete, but I can't PUT any
> > > commands to even delete data. I don't think there has ever been more than
> > > one node, but I really don't even know how to start more than one node. I
> > > only restart the ES server using the init commands as a service. I have 58
> > > unassigned shards, but I don't know how to correct them. I don't really
> > > need any of the data either, I just need to be Green again, so I can start
> > > over, but I would like to figure out what happened so I can not do it again
> > > when I do need the data.
> > >
> > > On Sunday, August 5, 2012 1:58:07 PM UTC-4, David Pilato wrote:
> > >
> > > > My opinion is that you created some indexes and some of them with no
> > > > replica.
> > > > You started more than one node in your LAN. Then, you shutdown one node.
> > > >
> > > > ES can not give you a green or yellow health as some of your documents
> > > > (index with no replica) can not be seen by ES.
> > > >
> > > > Is that what happened?
> > > >
> > > > David
> > > >
> > > > --
> > > >
> > > > Le 5 août 2012 à 19:26, browe a écrit :
> > > >
> > > > My situation is similar. I have not been able to resolve this or find
> > > > any solution online yet..
> > > > My cluster health:
> > > > {
> > > > "cluster\_name" : "elasticsearch",
> > > > "status" : "red",
> > > > "timed\_out" : false,
> > > > "number\_of\_nodes" : 1,
> > > > "number\_of\_data\_nodes" : 1,
> > > > "active\_primary\_shards" : 22,
> > > > "active\_shards" : 22,
> > > > "relocating\_shards" : 0,
> > > > "initializing\_shards" : 0,
> > > > "unassigned\_shards" : 58
> > > > }
> > > > I can do searches, but if I try an XPUT it fails. This did work at one
> > > > time.
> > > >
> > > > curl -XPUT '[http://localhost:9200/twitter/tweet/1](http://localhost:9200/twitter/tweet/1)' -d '{
> > > > "user" : "kimchy",
> > > > "post\_date" : "2009-11-15T14:12:12",
> > > > "message" : "trying out Elastic Search"
> > > > }'
> > > > {"error":"UnavailableShardsException[[twitter][2] [2] shardIt, [0]
> > > > active : Timeout waiting for [1m], request: index {[twitter][tweet][1],
> > > > source[{\n "user" : "kimchy",\n "post\_date" :
> > > > "2009-11-15T14:12:12",\n "message" : "trying out Elastic
> > > > Search"\n}]}]","status":503
> > > >
> > > > What also appears odd is that my failed PUTS don't show up in the log.
> > > > But when I shutdown and startup the server, those activities do show up in
> > > > the log. ( I deleted the old log to start over and try everything again)
> > > >
> > > > On Thursday, July 12, 2012 2:51:01 PM UTC-4, Yuhan wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I got "No Active Record" exception on some index but works for other.
> > > > > The health was indicating status=red, and there are unassigned\_shards.
> > > > >
> > > > > {
> > > > > "cluster\_name" : "elasticsearch",
> > > > > "status" : "red",
> > > > > "timed\_out" : false,
> > > > > "number\_of\_nodes" : 1,
> > > > > "number\_of\_data\_nodes" : 1,
> > > > > "active\_primary\_shards" : 54,
> > > > > "active\_shards" : 54,
> > > > > "relocating\_shards" : 0,
> > > > > "initializing\_shards" : 0,
> > > > > "unassigned\_shards" : 66
> > > > > }
> > > > >
> > > > > what should I do to correct this situration? could someone give some
> > > > > recommended reading?
> > > > >
> > > > > Thank you.
> > > > >
> > > > > Yuhan
---
### Author:  [@inZania](https://discuss.elastic.co/u/inZania)
#### Post date: [March 17, 2013, 9:39pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/11 "2013-03-17T21:39:06Z")
I know this is an old thread but I have to jump in:
I just had this happen on my production servers, where the status was red due to a system shutdown on one of my nodes, and I ended up being forced to delete 6GB of data in order to get the status to turn green again. Very frustrating ☹
---
### Author:  [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)
#### Post date: [March 18, 2013, 9:20am UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/12 "2013-03-18T09:20:42Z")
On Sun, 2013-03-17 at 14:39 -0700, inZania wrote:
> I know this is an old thread but I have to jump in:
>
> I just had this happen on my production servers, where the status was red
> due to a system shutdown on one of my nodes, and I ended up being forced to
> delete 6GB of data in order to get the status to turn green again. Very
> frustrating ☹
You shouldn't need to do this, but given that you haven't provided any
details about your cluster, or the problem that you saw, it's impossible
to provide advice
clint
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@utkarsh2012](https://discuss.elastic.co/u/utkarsh2012)
#### Post date: [April 2, 2013, 7:13pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/13 "2013-04-02T19:13:43Z")
I just stumbled on the same issue. I am evaluating and currently using ES
to index logs.
I started a new node by mistake with the same name so it formed a cluster.
I killed the new node but now the original node which indexes logs has
status=red
Is there a way I can fix this without deleting all that data?
Thanks,
-Utkarsh
On Monday, March 18, 2013 2:20:42 AM UTC-7, Clinton Gormley wrote:
> On Sun, 2013-03-17 at 14:39 -0700, inZania wrote:
>
> > I know this is an old thread but I have to jump in:
> >
> > I just had this happen on my production servers, where the status was
> > red
> > due to a system shutdown on one of my nodes, and I ended up being forced
> > to
> > delete 6GB of data in order to get the status to turn green again. Very
> > frustrating ☹
>
> You shouldn't need to do this, but given that you haven't provided any
> details about your cluster, or the problem that you saw, it's impossible
> to provide advice
>
> clint
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [April 2, 2013, 8:17pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/14 "2013-04-02T20:17:10Z")
Can you provide details about how you killed your node? If you killed
the process with SIGKILL, you may have damaged your data, but you do not
give much information about the state of your logs, your files, and your
cluster, so it's hard to give advice.
Jörg
Am 02.04.13 21:13, schrieb [utkarsh2012@gmail.com](mailto:utkarsh2012@gmail.com):
> I started a new node by mistake with the same name so it formed a
> cluster. I killed the new node but now the original node which indexes
> logs has status=red
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@utkarsh2012](https://discuss.elastic.co/u/utkarsh2012)
#### Post date: [April 2, 2013, 8:26pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/15 "2013-04-02T20:26:25Z")
I killed the extra node using SIGKILL.
I am running just one node of Elasticsearch for logstash. What kind of
information would help?
I got this error in logstash error log:
{:message=\>"Failed to index an event, will retry",
:exception=\>org.elasticsearch.action.UnavailableShardsException:
[logstash-2013.03.29][0] [2] shardIt, [0] active : Timeout waiting for
[1m],
request: index {...request..}
My original ES node has a bunch of indexes like: logstash-2013.03.25 and
the new node created a index (say myindex) which I don't really need, I was
just playing around with the other node, didn't expect it will
automatically discover the other node (which is cool!) since it had the
same default cluster name.
Thanks,
-Utkarsh
On Tue, Apr 2, 2013 at 1:17 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:
> Can you provide details about how you killed your node? If you killed the
> process with SIGKILL, you may have damaged your data, but you do not give
> much information about the state of your logs, your files, and your
> cluster, so it's hard to give advice.
>
> Jörg
>
> Am 02.04.13 21:13, schrieb [utkarsh2012@gmail.com](mailto:utkarsh2012@gmail.com):
>
> I started a new node by mistake with the same name so it formed a
>
> > cluster. I killed the new node but now the original node which indexes logs
> > has status=red
>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "elasticsearch" group.
> To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\*
> topic/elasticsearch/k-\*\*TYhXM8dXQ/unsubscribe?hl=en-US[https://groups.google.com/d/topic/elasticsearch/k-TYhXM8dXQ/unsubscribe?hl=en-US](https://groups.google.com/d/topic/elasticsearch/k-TYhXM8dXQ/unsubscribe?hl=en-US)
> \*\*.
> To unsubscribe from this group and all its topics, send an email to
> elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com)
> .
> For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out)
> .
--
Thanks,
-Utkarsh
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [April 2, 2013, 8:29pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/16 "2013-04-02T20:29:02Z")
This message is not from the Elasticsearch cluster, it's from logstash I
assume.
Jörg
Am 02.04.13 22:26, schrieb Utkarsh Sengar:
> I got this error in logstash error log:
> {:message=\>"Failed to index an event, will retry",
> :exception=\>org.elasticsearch.action.UnavailableShardsException:
> [logstash-2013.03.29][0] [2] shardIt, [0] active : Timeout waiting for
> [1m],
> request: index {...request..}
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@sujoysett](https://discuss.elastic.co/u/sujoysett)
#### Post date: [April 3, 2013, 6:36am UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/17 "2013-04-03T06:36:34Z")
Hi,
Your case sounds similar to some I faced several times.
By mistake started a new node, so at a certain instance there is one extra
node, and ES automatically starts moving and balancing data between the
nodes.
By the time I notice the extra node, some shards have already moved there,
and just shutting down the node might result in some shards being not
available, thus making the cluster red.
Two solution to the problem (which I follow):
1. keep the new node up - increase replica count to have a copy of each
shard in at least one node except this extra one - now shut down this node
- readjust replica count.
1. make the node up after assigning certain tag value in the node - issue
command to exclude shards from this tag - in some time shards will move out
from this node - shutdown the node.
Not sure whether the same problem has occurred in your case, just thought
of sharing in case it helps.
- Sujoy.
On Wednesday, April 3, 2013 12:43:43 AM UTC+5:30, [utkar...@gmail.com](mailto:utkar...@gmail.com) wrote:
> I just stumbled on the same issue. I am evaluating and currently using ES
> to index logs.
> I started a new node by mistake with the same name so it formed a cluster.
> I killed the new node but now the original node which indexes logs has
> status=red
>
> Is there a way I can fix this without deleting all that data?
>
> Thanks,
> -Utkarsh
>
> On Monday, March 18, 2013 2:20:42 AM UTC-7, Clinton Gormley wrote:
>
> > On Sun, 2013-03-17 at 14:39 -0700, inZania wrote:
> >
> > > I know this is an old thread but I have to jump in:
> > >
> > > I just had this happen on my production servers, where the status was
> > > red
> > > due to a system shutdown on one of my nodes, and I ended up being
> > > forced to
> > > delete 6GB of data in order to get the status to turn green again. Very
> > > frustrating ☹
> >
> > You shouldn't need to do this, but given that you haven't provided any
> > details about your cluster, or the problem that you saw, it's impossible
> > to provide advice
> >
> > clint
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Henley\_Wing](https://discuss.elastic.co/u/Henley_Wing)
#### Post date: [September 20, 2013, 8:12pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/18 "2013-09-20T20:12:19Z")
SO... there is a clean way to resolve this. Although I must say the
Elasticsearch documentation is very very confusing (all these buzzwords
like cluster and zen discovery boggles my mind!)
1.
Now, if you have 2 instances, one in port 9200, and the other in 9201. And
you want ALL the shards to be in 9200.
Run this command to disable allocation in the 9201 instance. You can change
persistent to transient if you want this change to not be permanent. I'd
keep it persistent so this doesn't ever happen again.
curl -XPUT localhost:9201/\_cluster/settings -d '{
"persistent" : {
"cluster.routing.allocation.disable\_allocation" : true
}
}'
1. Now, run the command to MOVE all the shards in the 9201 instance to 9200.
curl -XPOST 'localhost:9200/\_cluster/reroute' -d '{
"commands" : [ {
"move" :
{
"index" : "", "shard" : ,
"from\_node" : "\", "to\_node" : "\"
}
}
]
}'
You need to run this command for every shard in the 9201 instance (the one
you wanna get rid of).
That's it!
On Wednesday, April 3, 2013 2:36:34 AM UTC-4, Sujoy Sett wrote:
> Hi,
>
> Your case sounds similar to some I faced several times.
>
> By mistake started a new node, so at a certain instance there is one extra
> node, and ES automatically starts moving and balancing data between the
> nodes.
> By the time I notice the extra node, some shards have already moved there,
> and just shutting down the node might result in some shards being not
> available, thus making the cluster red.
>
> Two solution to the problem (which I follow):
>
> 1. keep the new node up - increase replica count to have a copy of each
> shard in at least one node except this extra one - now shut down this node
>
> - readjust replica count.
>
> 1. make the node up after assigning certain tag value in the node - issue
> command to exclude shards from this tag - in some time shards will move out
> from this node - shutdown the node.
>
> Not sure whether the same problem has occurred in your case, just thought
> of sharing in case it helps.
>
> - Sujoy.
>
> On Wednesday, April 3, 2013 12:43:43 AM UTC+5:30, utkar...@gmail.comwrote:
>
> > I just stumbled on the same issue. I am evaluating and currently using ES
> > to index logs.
> > I started a new node by mistake with the same name so it formed a
> > cluster. I killed the new node but now the original node which indexes logs
> > has status=red
> >
> > Is there a way I can fix this without deleting all that data?
> >
> > Thanks,
> > -Utkarsh
> >
> > On Monday, March 18, 2013 2:20:42 AM UTC-7, Clinton Gormley wrote:
> >
> > > On Sun, 2013-03-17 at 14:39 -0700, inZania wrote:
> > >
> > > > I know this is an old thread but I have to jump in:
> > > >
> > > > I just had this happen on my production servers, where the status was
> > > > red
> > > > due to a system shutdown on one of my nodes, and I ended up being
> > > > forced to
> > > > delete 6GB of data in order to get the status to turn green again.
> > > > Very
> > > > frustrating ☹
> > >
> > > You shouldn't need to do this, but given that you haven't provided any
> > > details about your cluster, or the problem that you saw, it's impossible
> > > to provide advice
> > >
> > > clint
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Jayanna\_Hallur](https://discuss.elastic.co/u/Jayanna_Hallur)
#### Post date: [March 31, 2017, 10:42pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/19 "2017-03-31T22:42:46Z")
Sujoy - if you are bringing up only 1 new node & you have primary & replication shards configured, bring down one node should not make the cluster state to RED..
You may you need to configure your ES nodes not store Primary & replica shards in the same node with the following config in yml file.. This may be issue in your case..
cluster.routing.allocation.same\_shard.host: true
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 10:01pm UTC](https://discuss.elastic.co/t/how-to-resolve-elasticsearch-status-red/8401/20 "2017-07-05T22:01:35Z")