# Cannot post new document to elasticsearch **URL:** \ **Category:** Elasticsearch\ **Created:** [March 27, 2012, 2:57pm UTC](https://discuss.elastic.co/t/cannot-post-new-document-to-elasticsearch/7144 "2012-03-27T14:57:58Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Gaurav\_Vijayvargiya](https://avatars.discourse-cdn.com/v4/letter/g/d6d6ee/32.png) [@Gaurav\_Vijayvargiya](https://discuss.elastic.co/u/Gaurav_Vijayvargiya)\ **Post date:** [March 27, 2012, 2:57pm UTC](https://discuss.elastic.co/t/cannot-post-new-document-to-elasticsearch/7144/1 "2012-03-27T14:57:58Z")
Hi We had a 5 node cluster (5 data nodes and 1 non-date master). One of the data nodes went down. Now when we try to post anything to the cluster it just hangs for ever. There is nothing in the logs. $ curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' { "cluster\_name" : "merchant\_service\_test", "status" : "yellow", "timed\_out" : false, "number\_of\_nodes" : 5, "number\_of\_data\_nodes" : 4, "active\_primary\_shards" : 20, "active\_shards" : 38, "relocating\_shards" : 0, "initializing\_shards" : 2, "unassigned\_shards" : 0 } $ curl -XPOST '[http://localhost:9200/m3/\_optimize](http://localhost:9200/m3/_optimize)' {"ok":true,"\_shards":{"total":40,"successful":37,"failed":1,"failures": [{"index":"m3","shard": 1,"reason":"BroadcastShardOperationFailedException[[m3][1] ]; nested: RemoteTransportException[[Arkon][inet[/10.188.214.210:9300]][indices/ optimize/shard]]; nested: FlushNotAllowedEngineException[[m3][1] Already flushing...]; "}]}} Its been like this for at least few hours now. I don't see much CPU/IO activity on any of the nodes. Any suggestions? ---
**Author:** ![Gaurav\_Vijayvargiya](https://avatars.discourse-cdn.com/v4/letter/g/d6d6ee/32.png) [@Gaurav\_Vijayvargiya](https://discuss.elastic.co/u/Gaurav_Vijayvargiya)\ **Post date:** [March 27, 2012, 5:21pm UTC](https://discuss.elastic.co/t/cannot-post-new-document-to-elasticsearch/7144/2 "2012-03-27T17:21:43Z")
Update, When I try to do a Get on an existing document, I alternately get it immediately or just wait for repsonse for ever. My number or replicas is 2. My guess is that one of the replicas is unavailable, thus when the query goes to that replica, it hangs. Whereas next time when the query goes to the other (good) replica, it returns immediately. Whereas for post it needs to write to both the replicas and thus it hangs. How can I fix the "initializing shards" issue? Gaurav On Mar 27, 7:57 am, Gaurav Vijayvargiya [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com) wrote: > Hi > > We had a 5 node cluster (5 data nodes and 1 non-date master). One of > the data nodes went down. Now when we try to post anything to the > cluster it just hangs for ever. There is nothing in the logs. > > $ curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' > { > "cluster\_name" : "merchant\_service\_test", > "status" : "yellow", > "timed\_out" : false, > "number\_of\_nodes" : 5, > "number\_of\_data\_nodes" : 4, > "active\_primary\_shards" : 20, > "active\_shards" : 38, > "relocating\_shards" : 0, > "initializing\_shards" : 2, > "unassigned\_shards" : 0 > > } > > $ curl -XPOST '[http://localhost:9200/m3/\_optimize](http://localhost:9200/m3/_optimize)' > {"ok":true,"\_shards":{"total":40,"successful":37,"failed":1,"failures": > [{"index":"m3","shard": > 1,"reason":"BroadcastShardOperationFailedException[[m3][1] ]; nested: > RemoteTransportException[[Arkon][inet[/10.188.214.210:9300]][indices/ > optimize/shard]]; nested: FlushNotAllowedEngineException[[m3][1] > Already flushing...]; "}]}} > > Its been like this for at least few hours now. I don't see much CPU/IO > activity on any of the nodes. > > Any suggestions? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 27, 2012, 6:54pm UTC](https://discuss.elastic.co/t/cannot-post-new-document-to-elasticsearch/7144/3 "2012-03-27T18:54:26Z")
Do you see any failures in the logs for those initializing shards? On Tue, Mar 27, 2012 at 7:21 PM, Gaurav Vijayvargiya \< [gvijayvargiya@gmail.com](mailto:gvijayvargiya@gmail.com)\> wrote: > Update, > > When I try to do a Get on an existing document, I alternately get it > immediately or just wait for repsonse for ever. > My number or replicas is 2. My guess is that one of the replicas is > unavailable, thus when the query goes to that replica, it hangs. > Whereas next time when the query goes to the other (good) replica, it > returns immediately. > > Whereas for post it needs to write to both the replicas and thus it > hangs. > > How can I fix the "initializing shards" issue? > > Gaurav > > On Mar 27, 7:57 am, Gaurav Vijayvargiya [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com) > wrote: > > > Hi > > > > We had a 5 node cluster (5 data nodes and 1 non-date master). One of > > the data nodes went down. Now when we try to post anything to the > > cluster it just hangs for ever. There is nothing in the logs. > > > > $ curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' > > { > > "cluster\_name" : "merchant\_service\_test", > > "status" : "yellow", > > "timed\_out" : false, > > "number\_of\_nodes" : 5, > > "number\_of\_data\_nodes" : 4, > > "active\_primary\_shards" : 20, > > "active\_shards" : 38, > > "relocating\_shards" : 0, > > "initializing\_shards" : 2, > > "unassigned\_shards" : 0 > > > > } > > > > $ curl -XPOST '[http://localhost:9200/m3/\_optimize](http://localhost:9200/m3/_optimize)' > > {"ok":true,"\_shards":{"total":40,"successful":37,"failed":1,"failures": > > [{"index":"m3","shard": > > 1,"reason":"BroadcastShardOperationFailedException[[m3][1] ]; nested: > > RemoteTransportException[[Arkon][inet[/10.188.214.210:9300]][indices/ > > optimize/shard]]; nested: FlushNotAllowedEngineException[[m3][1] > > Already flushing...]; "}]}} > > > > Its been like this for at least few hours now. I don't see much CPU/IO > > activity on any of the nodes. > > > > Any suggestions? ---
**Author:** ![Gaurav\_Vijayvargiya](https://avatars.discourse-cdn.com/v4/letter/g/d6d6ee/32.png) [@Gaurav\_Vijayvargiya](https://discuss.elastic.co/u/Gaurav_Vijayvargiya)\ **Post date:** [March 27, 2012, 8:06pm UTC](https://discuss.elastic.co/t/cannot-post-new-document-to-elasticsearch/7144/4 "2012-03-27T20:06:55Z")
Looked at the logs, there is nothing I could interpret. Btw, both the failing (initializing) shard were on the same node. I shut down that node and it recovered. So the good news is that the cluster is up and running now and so far all the data seems to be there. But I still don't know what caused the problem? ☹ On Mar 27, 11:54 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Do you see any failures in the logs for those initializing shards? > > On Tue, Mar 27, 2012 at 7:21 PM, Gaurav Vijayvargiya \< > > [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com)\> wrote: > > > Update, > > > When I try to do a Get on an existing document, I alternately get it > > immediately or just wait for repsonse for ever. > > My number or replicas is 2. My guess is that one of the replicas is > > unavailable, thus when the query goes to that replica, it hangs. > > Whereas next time when the query goes to the other (good) replica, it > > returns immediately. > > > Whereas for post it needs to write to both the replicas and thus it > > hangs. > > > How can I fix the "initializing shards" issue? > > > Gaurav > > > On Mar 27, 7:57 am, Gaurav Vijayvargiya [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com) > > wrote: > > > > > Hi > > > > We had a 5 node cluster (5 data nodes and 1 non-date master). One of > > > the data nodes went down. Now when we try to post anything to the > > > cluster it just hangs for ever. There is nothing in the logs. > > > > $ curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' > > > { > > > "cluster\_name" : "merchant\_service\_test", > > > "status" : "yellow", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 5, > > > "number\_of\_data\_nodes" : 4, > > > "active\_primary\_shards" : 20, > > > "active\_shards" : 38, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 2, > > > "unassigned\_shards" : 0 > > > > } > > > > $ curl -XPOST '[http://localhost:9200/m3/\_optimize](http://localhost:9200/m3/_optimize)' > > > {"ok":true,"\_shards":{"total":40,"successful":37,"failed":1,"failures": > > > [{"index":"m3","shard": > > > 1,"reason":"BroadcastShardOperationFailedException[[m3][1] ]; nested: > > > RemoteTransportException[[Arkon][inet[/10.188.214.210:9300]][indices/ > > > optimize/shard]]; nested: FlushNotAllowedEngineException[[m3][1] > > > Already flushing...]; "}]}} > > > > Its been like this for at least few hours now. I don't see much CPU/IO > > > activity on any of the nodes. > > > > Any suggestions? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 28, 2012, 10:31am UTC](https://discuss.elastic.co/t/cannot-post-new-document-to-elasticsearch/7144/5 "2012-03-28T10:31:58Z")
Strange..., which version are you using? On Tue, Mar 27, 2012 at 10:06 PM, Gaurav Vijayvargiya \< [gvijayvargiya@gmail.com](mailto:gvijayvargiya@gmail.com)\> wrote: > Looked at the logs, there is nothing I could interpret. > > Btw, both the failing (initializing) shard were on the same node. I > shut down that node and it recovered. So the good news is that the > cluster is up and running now and so far all the data seems to be > there. > > But I still don't know what caused the problem? ☹ > > On Mar 27, 11:54 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > Do you see any failures in the logs for those initializing shards? > > > > On Tue, Mar 27, 2012 at 7:21 PM, Gaurav Vijayvargiya \< > > > > [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com)\> wrote: > > > > > Update, > > > > > When I try to do a Get on an existing document, I alternately get it > > > immediately or just wait for repsonse for ever. > > > My number or replicas is 2. My guess is that one of the replicas is > > > unavailable, thus when the query goes to that replica, it hangs. > > > Whereas next time when the query goes to the other (good) replica, it > > > returns immediately. > > > > > Whereas for post it needs to write to both the replicas and thus it > > > hangs. > > > > > How can I fix the "initializing shards" issue? > > > > > Gaurav > > > > > On Mar 27, 7:57 am, Gaurav Vijayvargiya [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com) > > > wrote: > > > > > > > Hi > > > > > > We had a 5 node cluster (5 data nodes and 1 non-date master). One of > > > > the data nodes went down. Now when we try to post anything to the > > > > cluster it just hangs for ever. There is nothing in the logs. > > > > > > $ curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' > > > > { > > > > "cluster\_name" : "merchant\_service\_test", > > > > "status" : "yellow", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 5, > > > > "number\_of\_data\_nodes" : 4, > > > > "active\_primary\_shards" : 20, > > > > "active\_shards" : 38, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 2, > > > > "unassigned\_shards" : 0 > > > > > > } > > > > > > $ curl -XPOST '[http://localhost:9200/m3/\_optimize](http://localhost:9200/m3/_optimize)' > > {"ok":true,"\_shards":{"total":40,"successful":37,"failed":1,"failures": > > > > > [{"index":"m3","shard": > > > > 1,"reason":"BroadcastShardOperationFailedException[[m3][1] ]; nested: > > > > RemoteTransportException[[Arkon][inet[/10.188.214.210:9300 > > > > ]][indices/ > > > > optimize/shard]]; nested: FlushNotAllowedEngineException[[m3][1] > > > > Already flushing...]; "}]}} > > > > > > Its been like this for at least few hours now. I don't see much > > > > CPU/IO > > > > activity on any of the nodes. > > > > > > Any suggestions? ---
**Author:** ![Gaurav\_Vijayvargiya](https://avatars.discourse-cdn.com/v4/letter/g/d6d6ee/32.png) [@Gaurav\_Vijayvargiya](https://discuss.elastic.co/u/Gaurav_Vijayvargiya)\ **Post date:** [March 28, 2012, 5:15pm UTC](https://discuss.elastic.co/t/cannot-post-new-document-to-elasticsearch/7144/6 "2012-03-28T17:15:38Z")
elasticsearch-0.18.7 On Mar 28, 3:31 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Strange..., which version are you using? > > On Tue, Mar 27, 2012 at 10:06 PM, Gaurav Vijayvargiya \< > > [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com)\> wrote: > > > Looked at the logs, there is nothing I could interpret. > > > Btw, both the failing (initializing) shard were on the same node. I > > shut down that node and it recovered. So the good news is that the > > cluster is up and running now and so far all the data seems to be > > there. > > > But I still don't know what caused the problem? ☹ > > > On Mar 27, 11:54 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > Do you see any failures in the logs for those initializing shards? > > > > On Tue, Mar 27, 2012 at 7:21 PM, Gaurav Vijayvargiya \< > > > > [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com)\> wrote: > > > > > > > Update, > > > > > When I try to do a Get on an existing document, I alternately get it > > > > immediately or just wait for repsonse for ever. > > > > My number or replicas is 2. My guess is that one of the replicas is > > > > unavailable, thus when the query goes to that replica, it hangs. > > > > Whereas next time when the query goes to the other (good) replica, it > > > > returns immediately. > > > > > Whereas for post it needs to write to both the replicas and thus it > > > > hangs. > > > > > How can I fix the "initializing shards" issue? > > > > > Gaurav > > > > > On Mar 27, 7:57 am, Gaurav Vijayvargiya [gvijayvarg...@gmail.com](mailto:gvijayvarg...@gmail.com) > > > > wrote: > > > > > > > > > Hi > > > > > > We had a 5 node cluster (5 data nodes and 1 non-date master). One of > > > > > the data nodes went down. Now when we try to post anything to the > > > > > cluster it just hangs for ever. There is nothing in the logs. > > > > > > $ curl -XGET '[http://localhost:9200/\_cluster/health?pretty=true](http://localhost:9200/_cluster/health?pretty=true)' > > > > > { > > > > > "cluster\_name" : "merchant\_service\_test", > > > > > "status" : "yellow", > > > > > "timed\_out" : false, > > > > > "number\_of\_nodes" : 5, > > > > > "number\_of\_data\_nodes" : 4, > > > > > "active\_primary\_shards" : 20, > > > > > "active\_shards" : 38, > > > > > "relocating\_shards" : 0, > > > > > "initializing\_shards" : 2, > > > > > "unassigned\_shards" : 0 > > > > > > } > > > > > > $ curl -XPOST '[http://localhost:9200/m3/\_optimize](http://localhost:9200/m3/_optimize)' > > > {"ok":true,"\_shards":{"total":40,"successful":37,"failed":1,"failures": > > > > > > > [{"index":"m3","shard": > > > > > 1,"reason":"BroadcastShardOperationFailedException[[m3][1] ]; nested: > > > > > RemoteTransportException[[Arkon][inet[/10.188.214.210:9300 > > > > > ]][indices/ > > > > > optimize/shard]]; nested: FlushNotAllowedEngineException[[m3][1] > > > > > Already flushing...]; "}]}} > > > > > > Its been like this for at least few hours now. I don't see much > > > > > CPU/IO > > > > > activity on any of the nodes. > > > > > > Any suggestions? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:34am UTC](https://discuss.elastic.co/t/cannot-post-new-document-to-elasticsearch/7144/7 "2017-07-06T03:34:18Z")