# Shard in initialing state for a long period of time now **URL:** \ **Category:** Logstash\ **Created:** [February 22, 2016, 1:09am UTC](https://discuss.elastic.co/t/shard-in-initialing-state-for-a-long-period-of-time-now/42347 "2016-02-22T01:09:07Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![James\_Tang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_tang/32/8112_2.png) [@James\_Tang](https://discuss.elastic.co/u/James_Tang)\ **Post date:** [February 22, 2016, 1:09am UTC](https://discuss.elastic.co/t/shard-in-initialing-state-for-a-long-period-of-time-now/42347/1 "2016-02-22T01:09:07Z")
Cluster ID : 45cd54 Shard location-541275-1602 Observing KOPF plugin show that the shard is in an initializing state for more than 10 minutes now. Please restart or do your magic. I have stop all indexing activity and all known connection to it but it still did not auto recover. I think it is also holding up the snapshot job with this error: `[2016-02-22 00:48:34,686][INFO][rest.suppressed] /\_status Params: {index=\_status, recovery=true} java.lang.IllegalArgumentException: No feature for name [\_status] at org.elasticsearch.action.admin.indices.get.GetIndexRequest$Feature.fromName(GetIndexRequest.java:82) at . . ---
**Author:** ![alexbrasetvik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexbrasetvik/32/15538_2.png) [@alexbrasetvik](https://discuss.elastic.co/u/alexbrasetvik)\ **Post date:** [February 22, 2016, 3:26am UTC](https://discuss.elastic.co/t/shard-in-initialing-state-for-a-long-period-of-time-now/42347/2 "2016-02-22T03:26:52Z")
Hi, The problem with this shard is likely due to this event in your logs: `[2016-02-21 18:08:29,172][WARN][index.engine] [instance-0000000009] [location-541275-1602][0] failed engine [merge failed] org.apache.lucene.index.MergePolicy$MergeException: java.lang.OutOfMemoryError: Direct buffer memory at org.e …` The OutOfMemory-event should normally trigger a node-restart, and we're looking into why that does not happen. I've restarted your node. To avoid this OOM-events at all, which any production cluster must, you'll need to reduce your memory requirements and/or upgrade the cluster. Production clusters should also use the high availability offerings, so there's replicas in multiple availability zones. The memory pressure indicator shows the _current_ memory pressure. Your OOM seems to be caused by a sudden increase, e.g. due to overwhelming the cluster with concurrent index requests that queue up. ---
**Author:** ![James\_Tang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_tang/32/8112_2.png) [@James\_Tang](https://discuss.elastic.co/u/James_Tang)\ **Post date:** [February 22, 2016, 4:23am UTC](https://discuss.elastic.co/t/shard-in-initialing-state-for-a-long-period-of-time-now/42347/3 "2016-02-22T04:23:38Z")
Noted. Was trying to read from the indices for reindexing the docs. ---
**Author:** ![James\_Tang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_tang/32/8112_2.png) [@James\_Tang](https://discuss.elastic.co/u/James_Tang)\ **Post date:** [February 22, 2016, 8:38am UTC](https://discuss.elastic.co/t/shard-in-initialing-state-for-a-long-period-of-time-now/42347/4 "2016-02-22T08:38:39Z")
Hmm. I just realized a query which you may help me with understanding. In the event that the logstash elasticsearch input plugin hit an issue; (For my case "Error: Unable to establish loopback connection"), will it clear the scroll? What happens when the plugin restart? Does it restarts or continue? Can restarting of the plugin contribute to the additional memory utilization which eventually lead to the OOM exception? ---
**Author:** ![beiske](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beiske/32/4042_2.png) [@beiske](https://discuss.elastic.co/u/beiske)\ **Post date:** [February 22, 2016, 6:19pm UTC](https://discuss.elastic.co/t/shard-in-initialing-state-for-a-long-period-of-time-now/42347/5 "2016-02-22T18:19:21Z")
This is an excellent question for the Logstash category Best regards Konrad Beiske Software Engineer Elastic ---
**Author:** ![James\_Tang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/james_tang/32/8112_2.png) [@James\_Tang](https://discuss.elastic.co/u/James_Tang)\ **Post date:** [February 23, 2016, 2:32am UTC](https://discuss.elastic.co/t/shard-in-initialing-state-for-a-long-period-of-time-now/42347/6 "2016-02-23T02:32:34Z")
This is my setup: * * * ``` input { elasticsearch { [blah blah] docinfo => true type => "tcpdump_final" } } filter { metrics { meter => "events" add_tag => "metric" } } output { if [type] == "tcpdump_final" { elasticsearch { } } if "metric" in [tags] { stdout { codec => line { format => "rate: %{[events][rate_1m]} ::: count %{[events][count]}" } } } } ``` Upon hitting an error with the input, the plugin restarts and your search query get executed again. The important thing to note is that you are going to be assigned a new scroll\_id. I derived at this conclusion from the metric filter printout showing the count \> than the original quantity of document. It has to restart for the count to exceed the total number of document in the original. 2nd observation which is the important one is from querying: > GET /\_nodes/stats/indices/search?pretty The search context keeps on increasing. I believe this contributes to my elasticsearch instance hitting the OOM exception. (Left it running overnight. Never expected that.) The behaviors makes it tough for those with limited resources to perform reindexing using logstash. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:10am UTC](https://discuss.elastic.co/t/shard-in-initialing-state-for-a-long-period-of-time-now/42347/7 "2017-07-06T05:10:11Z")
Closed on July 6, 2017, 5:10am UTC