# All commands returning - failed to process cluster event (503) **URL:** \ **Category:** Elasticsearch\ **Created:** [June 23, 2014, 11:01pm UTC](https://discuss.elastic.co/t/all-commands-returning-failed-to-process-cluster-event-503/18276 "2014-06-23T23:01:25Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Ashish\_Nigam\_3](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@Ashish\_Nigam\_3](https://discuss.elastic.co/u/Ashish_Nigam_3)\ **Post date:** [June 23, 2014, 11:01pm UTC](https://discuss.elastic.co/t/all-commands-returning-failed-to-process-cluster-event-503/18276/1 "2014-06-23T23:01:25Z")
Hi, Production cluster is in yellow state for more than a day now. I do see that unassigned shards are decreasing, but at a very slow pace. (600 shards in last 5 hours) So, I decided to remove very old indexes and also allocate unassigned shards using reroute command. Unfortunately, all commands are returning with this error - { "error" : "RemoteTransportException[[search00][inet[/10.0.1.10:9300]][cluster/reroute]]; nested: ProcessClusterEventTimeoutException[failed to process cluster event (cluster\_reroute (api)) within 30s]; ", "status" : 503 } Why I am getting this error consistently? And any idea how to speed up recovery and also be able to execute commands? Another data point is that cluster went to yellow state as one node went out of the cluster and then joined back after sometime. Thanks Ashish -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANsfGRmbw%3DAS3txAZA2O5PrK-oC52LysswDuyA2%3DgubydE74rA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANsfGRmbw%3DAS3txAZA2O5PrK-oC52LysswDuyA2%3DgubydE74rA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ashish\_Nigam\_3](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@Ashish\_Nigam\_3](https://discuss.elastic.co/u/Ashish_Nigam_3)\ **Post date:** [June 23, 2014, 11:50pm UTC](https://discuss.elastic.co/t/all-commands-returning-failed-to-process-cluster-event-503/18276/2 "2014-06-23T23:50:05Z")
Here's sample error message that I see for unassigned shards - [2014-06-23 04:59:08,087][WARN][cluster.action.shard] [search02] sending failed shard for [an\_shnindex\_9871\_2014\_1][3], node[bc03gt7WRx-XYToz7nHVqg], [P], s[STARTED], reason [master [search00][PDXvmuL\_TpG7bEIfIUvBDg][inet[/10.0.1.10:9300]]{master=true, zone=zone\_spop-sjc} marked shard as started, but shard has not been created, mark shard as failed] On Mon, Jun 23, 2014 at 4:01 PM, Ashish Nigam [ashnigamtech@gmail.com](mailto:ashnigamtech@gmail.com) wrote: > Hi, > Production cluster is in yellow state for more than a day now. I do see > that unassigned shards are decreasing, but at a very slow pace. (600 shards > in last 5 hours) > So, I decided to remove very old indexes and also allocate unassigned > shards using reroute command. Unfortunately, all commands are returning > with this error - > > { > > "error" : "RemoteTransportException[[search00][inet[/10.0.1.10:9300]][cluster/reroute]]; > nested: ProcessClusterEventTimeoutException[failed to process cluster event > (cluster\_reroute (api)) within 30s]; ", > > "status" : 503 > } > > Why I am getting this error consistently? And any idea how to speed up > recovery and also be able to execute commands? > > Another data point is that cluster went to yellow state as one node went > out of the cluster and then joined back after sometime. > > Thanks > Ashish -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CANsfGRk817E2VzotntG3UeijGo%2BKMpC%3DyY1Guj9sTUf1Nkx3Fw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CANsfGRk817E2VzotntG3UeijGo%2BKMpC%3DyY1Guj9sTUf1Nkx3Fw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:20am UTC](https://discuss.elastic.co/t/all-commands-returning-failed-to-process-cluster-event-503/18276/3 "2017-07-06T01:20:23Z")