# Suspect illegal state: trying to move shard from primary mode to replica mode **URL:** \ **Category:** Elasticsearch\ **Created:** [June 17, 2015, 1:55pm UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845 "2015-06-17T13:55:09Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\ **Post date:** [June 17, 2015, 1:55pm UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/1 "2015-06-17T13:55:09Z")
Hi all, I have a 6 node cluster in production running ES 1.5.0, and I'm seeing these messages roughly 7 times a day across all machines in the cluster. I did some searching, and was not able to find an explanation of why this might be happening, and what I could do to fix it. The shards that are in error are different across all machines, if that helps. The one thread I found mentioned making sure that: discovery.zen.minimum\_master\_nodes is set appropriately, which mine is! All other mentions of this error are for some really old versions of ES, so I'm at a bit of a loss. Might anyone have a suggestion? Many thanks, Chris ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 18, 2015, 4:26am UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/2 "2015-06-18T04:26:49Z")
Is anything else happening around that time, index creation or something else? ---
**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\ **Post date:** [June 18, 2015, 1:20pm UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/3 "2015-06-18T13:20:02Z")
Thanks Mark. Not index creation, but definitely document indexing is going on. No searches to speak of either. The index rate is pretty heavy, as I do see the "now throttling indexing" messages for other indexes, not the ones that are reported to be in illegal states. Sorry that's not much to go on. Chris ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 18, 2015, 9:56pm UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/4 "2015-06-18T21:56:08Z")
Can you paste a few of the logs lines that contain this, as well as the lines around it for context (if there's any). ---
**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\ **Post date:** [June 18, 2015, 11:40pm UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/5 "2015-06-18T23:40:36Z")
Certainly! Here's one: ``` [2015-06-18 10:47:28,695][INFO][index.engine] [elasticsearch-bdprodes02] [derbysoft-agoda-20150618][0] stop throttling indexing: numMergesInFlight=2, maxNumMerges=3 [2015-06-18 10:47:38,475][INFO][index.engine] [elasticsearch-bdprodes02] [derbysoft-carlson-20150618][1] now throttling indexing: numMergesInFlight=4, maxNumMerges=3 [2015-06-18 10:47:38,529][INFO][index.engine] [elasticsearch-bdprodes02] [derbysoft-carlson-20150618][1] stop throttling indexing: numMergesInFlight=2, maxNumMerges=3 [2015-06-18 11:28:52,276][WARN][index.shard] [elasticsearch-bdprodes02] [derbysoft-apache-20150612][1] suspect illegal state: trying to move shard from primary mode to replica mode [2015-06-18 11:44:35,129][INFO][index.engine] [elasticsearch-bdprodes02] [derbysoft-agoda-20150618][0] now throttling indexing: numMergesInFlight=4, maxNumMerges=3 [2015-06-18 11:44:35,653][INFO][index.engine] [elasticsearch-bdprodes02] [derbysoft-agoda-20150618][0] stop throttling indexing: numMergesInFlight=2, maxNumMerges=3 ``` and another: ``` [2015-06-17 00:40:55,260][INFO][cluster.metadata] [elasticsearch-bdprodes01] [derbysoft-agoda-20150617] update_mapping [http_access_dwarf4agodaplugin] (dynamic) [2015-06-17 00:49:02,716][WARN][index.shard] [elasticsearch-bdprodes01] [derbysoft-apache-20150615][1] suspect illegal state: trying to move shard from primary mode to replica mode [2015-06-17 00:52:19,905][INFO][index.engine] [elasticsearch-bdprodes01] [derbysoft-bestwestern-20150617][0] now throttling indexing: numMergesInFlight=4, maxNumMerges=3 ``` and one more: ``` [2015-06-17 07:47:28,336][INFO][index.engine] [elasticsearch-bdprodes01] [derbysoft-hilton-20150616][1] stop throttling indexing: numMergesInFlight=2, maxNumMerges=3 [2015-06-17 07:47:29,305][INFO][index.engine] [elasticsearch-bdprodes01] [derbysoft-hilton-20150616][1] now throttling indexing: numMergesInFlight=4, maxNumMerges=3 [2015-06-17 07:47:29,395][INFO][index.engine] [elasticsearch-bdprodes01] [derbysoft-hilton-20150616][1] stop throttling indexing: numMergesInFlight=2, maxNumMerges=3 [2015-06-17 07:47:44,288][INFO][index.engine] [elasticsearch-bdprodes01] [derbysoft-hilton-20150615][0] now throttling indexing: numMergesInFlight=4, maxNumMerges=3 [2015-06-17 07:47:44,357][INFO][index.engine] [elasticsearch-bdprodes01] [derbysoft-hilton-20150615][0] stop throttling indexing: numMergesInFlight=2, maxNumMerges=3 [2015-06-17 07:48:06,648][WARN][index.shard] [elasticsearch-bdprodes01] [derbysoft-apache-20150617][1] suspect illegal state: trying to move shard from primary mode to replica mode [2015-06-17 07:57:50,387][INFO][cluster.metadata] [elasticsearch-bdprodes01] [derbysoft-hilton-20150617] update_mapping [hotel_avail_hiltonplugin4agoda] (dynamic) [2015-06-17 07:58:52,801][INFO][index.engine] [elasticsearch-bdprodes01] [derbysoft-hilton-20150617][1] now throttling indexing: numMergesInFlight=4, maxNumMerges=3 ``` Thank you again. Chris ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 19, 2015, 12:35am UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/6 "2015-06-19T00:35:34Z")
[https://github.com/elastic/elasticsearch/issues/11395](https://github.com/elastic/elasticsearch/issues/11395) may be related. Can you you try [reindexing one of those indices](https://gist.github.com/markwalkom/8a7201e3f6ea4354ae06) to see if it helps? If it does we can then look at getting more information around why this is happening. ---
**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\ **Post date:** [June 19, 2015, 1:25am UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/7 "2015-06-19T01:25:06Z")
Absolutely. Once I run the re-indexer, how to I test the new index to see if anything changed? Chris ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 19, 2015, 1:26am UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/8 "2015-06-19T01:26:35Z")
The errors for that index should stop, it's a bit of a "hammer for a bug" approach, but it's a start. ---
**Author:** ![Chris\_Neal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_neal/32/3527_2.png) [@Chris\_Neal](https://discuss.elastic.co/u/Chris_Neal)\ **Post date:** [June 24, 2015, 4:11pm UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/9 "2015-06-24T16:11:13Z")
Just wanted to follow up on this. I spent several days dealing with some stability problems, and I'm getting back to this now. For whatever reason, these messages have stopped. I haven't had any in 5 days now, so I'm holding off on re-indexing things for now. Thanks Mark for your help, and if these come back, I'll know what to try! Chris ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:05am UTC](https://discuss.elastic.co/t/suspect-illegal-state-trying-to-move-shard-from-primary-mode-to-replica-mode/23845/10 "2017-07-06T00:05:35Z")