# Elasticsearch is in red state: class org.apache.lucene.store.BufferedChecksumIndexInput cannot seek backwards **URL:** \ **Category:** Elasticsearch\ **Created:** [June 28, 2017, 9:03am UTC](https://discuss.elastic.co/t/elasticsearch-is-in-red-state-class-org-apache-lucene-store-bufferedchecksumindexinput-cannot-seek-backwards/91067 "2017-06-28T09:03:39Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![dipathak](https://avatars.discourse-cdn.com/v4/letter/d/3ab097/32.png) [@dipathak](https://discuss.elastic.co/u/dipathak)\ **Post date:** [June 28, 2017, 9:03am UTC](https://discuss.elastic.co/t/elasticsearch-is-in-red-state-class-org-apache-lucene-store-bufferedchecksumindexinput-cannot-seek-backwards/91067/1 "2017-06-28T09:03:39Z")
Hi, We are using Elasticsearch 5.2 on a 3 node cluster. Today, ES went into red state and it does not seem to be recovering from it. Can someone please take a look. What could have caused this state? Do we have to reroute shard manually, with possibly data loss. { "index" : "cfileindex", "shard" : 8, "primary" : true, "current\_state" : "unassigned", "unassigned\_info" : { "reason" : "ALLOCATION\_FAILED", "at" : "2017-06-25T11:15:18.550Z", "failed\_allocation\_attempts" : 5, "details" : "failed recovery, failure RecoveryFailedException[[cfileindex][8]: Recovery failed on {14038005774639}{ui1ui89bSzmlBCQ0qD7SiQ}{f-LtGPOSSk2X-lljpAL0DA}{10.2.32.169}{10.2.32.169:25800}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[class org.apache.lucene.store.BufferedChecksumIndexInput cannot seek backwards (pos=-16 getFilePointer()=0)]; ", "last\_allocation\_status" : "no" }, "can\_allocate" : "no", "allocate\_explanation" : "cannot allocate because allocation is not permitted to any of the nodes that hold an in-sync shard copy", "node\_allocation\_decisions" : [ { "node\_id" : "ui1ui89bSzmlBCQ0qD7SiQ", "node\_name" : "14038005774639", "transport\_address" : "10.2.32.169:25800", "node\_decision" : "no", "store" : { "in\_sync" : true, "allocation\_id" : "sQWGfvJIRwqWn2eNjVn3pg" }, "deciders" : [ { "decider" : "max\_retry", "decision" : "NO", "explanation" : "shard has exceeded the maximum number of retries [5] on failed allocation attempts - manually call [/\_cluster/reroute?retry\_failed=true] to retry, [unassigned\_info[[reason=ALLOCATION\_FAILED], at[2017-06-25T11:15:18.550Z], failed\_attempts[5], delayed=false, details[failed recovery, failure RecoveryFailedException[[cfileindex][8]: Recovery failed on {14038005774639}{ui1ui89bSzmlBCQ0qD7SiQ}{f-LtGPOSSk2X-lljpAL0DA}{10.2.32.169}{10.2.32.169:25800}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[class org.apache.lucene.store.BufferedChecksumIndexInput cannot seek backwards (pos=-16 getFilePointer()=0)]; ], allocation\_status[deciders\_no]]]" } ] } ] } ======================================== Information in log files. [2017-06-28T00:23:27,554][WARN][o.e.c.a.s.ShardStateAction] [14038005774639] [cfileindex][6] received shard failed for shard id [[cfileindex][6]], allocation id [zT9iw10oRrmasz-icwRqFA], primary term [578], message [mark copy as stale] [2017-06-28T00:23:27,568][WARN][o.e.c.a.s.ShardStateAction] [14038005774639] [cfileindex][9] received shard failed for shard id [[cfileindex][9]], allocation id [WFJT1sjzT9-dQiREAD8rXA], primary term [573], message [mark copy as stale] [2017-06-28T00:23:27,645][WARN][o.e.c.a.s.ShardStateAction] [14038005774639] [cfileindex][4] received shard failed for shard id [[cfileindex][4]], allocation id [WFRVAMQ\_SLCxisSkjbTOTQ], primary term [573], message [mark copy as stale] [2017-06-28T00:23:27,682][WARN][o.e.c.a.s.ShardStateAction] [14038005774639] [cfileindex][0] received shard failed for shard id [[cfileindex][0]], allocation id [Wdrl7VAaSy2\_OXPUuJBeVQ], primary term [554], message [mark copy as stale] [2017-06-28T00:23:27,778][WARN][o.e.c.a.s.ShardStateAction] [14038005774639] [cfileindex][7] received shard failed for shard id [[cfileindex][7]], allocation id [N5MphZcWRq63voJ7\_ZhzEA], primary term [571], message [mark copy as stale] [2017-06-28T00:23:43,843][INFO][o.e.c.r.DelayedAllocationService] [14038005774639] scheduling reroute for delayed shards in [27.4s] (13 delayed shards) [2017-06-28T00:36:04,617][DEBUG][o.e.a.a.c.a.TransportClusterAllocationExplainAction] [14038005774639] explaining the allocation for [ClusterAllocationExplainRequest[index=cfileindex,shard=0,primary?=true,includeYesDecisions?=false], found shard [[cfileindex][0], node[ui1ui89bSzmlBCQ0qD7SiQ], [P], s[STARTED], a[id=4hc6JCLFSM2R1IAHU\_y2vg]] [2017-06-28T00:36:41,031][DEBUG][o.e.a.a.c.a.TransportClusterAllocationExplainAction] [14038005774639] explaining the allocation for [ClusterAllocationExplainRequest[index=cfileindex,shard=8,primary?=true,includeYesDecisions?=false], found shard [[cfileindex][8], node[null], [P], recovery\_source[existing recovery], s[UNASSIGNED], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2017-06-25T11:15:18.550Z], failed\_attempts[5], delayed=false, details[failed recovery, failure RecoveryFailedException[[cfileindex][8]: Recovery failed on {14038005774639}{ui1ui89bSzmlBCQ0qD7SiQ}{f-LtGPOSSk2X-lljpAL0DA}{10.2.32.169}{10.2.32.169:25800}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[class org.apache.lucene.store.BufferedChecksumIndexInput cannot seek backwards (pos=-16 getFilePointer()=0)]; ], allocation\_status[deciders\_no]]] [2017-06-28T00:37:41,122][DEBUG][o.e.a.a.c.a.TransportClusterAllocationExplainAction] [14038005774639] explaining the allocation for [ClusterAllocationExplainRequest[index=cauditindex,shard=1,primary?=true,includeYesDecisions?=false], found shard [[cauditindex][1], node[null], [P], recovery\_source[existing recovery], s[UNASSIGNED], unassigned\_info[[reason=ALLOCATION\_FAILED], at[2017-06-25T11:15:18.550Z], failed\_attempts[5], delayed=false, details[failed recovery, failure RecoveryFailedException[[cauditindex][1]: Recovery failed on {14038005774639}{ui1ui89bSzmlBCQ0qD7SiQ}{f-LtGPOSSk2X-lljpAL0DA}{10.2.32.169}{10.2.32.169:25800}]; nested: IndexShardRecoveryException[failed recovery]; nested: IllegalStateException[class org.apache.lucene.store.BufferedChecksumIndexInput cannot seek backwards (pos=-16 getFilePointer()=0)]; ], allocation\_status[deciders\_no]]] [2017-06-28T00:41:26,201][INFO][o.e.c.s.ClusterService] [14038005774639] removed {{14038005774618}{uI-QPxa6TWmUUTE-xoRc7Q}{X\_Pnry62Q8aCwz\_9kIWscQ}{10.2.32.165}{10.2.32.165:25800},}, reason: zen-disco-node-left({14038005774618}{uI-QPxa6TWmUUTE-xoRc7Q}{X\_Pnry62Q8aCwz\_9kIWscQ}{10.2.32.165}{10.2.32.165:25800}), reason(left)[{14038005774618}{uI-QPxa6TWmUUTE-xoRc7Q}{X\_Pnry62Q8aCwz\_9kIWscQ}{10.2.32.165}{10.2.32.165:25800} left] [2017-06-28T00:41:26,214][INFO][o.e.c.r.DelayedAllocationService] [14038005774639] scheduling reroute for delayed shards in [59.9s] (14 delayed shards) [2017-06-28T00:41:27,054][WARN][o.e.c.a.s.ShardStateAction] [14038005774639] [cfileindex][2] received shard failed for shard id [[cfileindex][2]], allocation id [Mzm\_bj5RTe-4fNXqAzk1oQ], primary term [565], message [mark copy as stale] [2017-06-28T00:41:27,054][WARN][o.e.c.a.s.ShardStateAction] [14038005774639] [cfileindex][1] received shard failed for shard id [[cfileindex][1]], allocation id [yqAH7hoOQz2lQUJnExpAJw], primary term [557], message [mark copy as stale] ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 26, 2017, 9:03am UTC](https://discuss.elastic.co/t/elasticsearch-is-in-red-state-class-org-apache-lucene-store-bufferedchecksumindexinput-cannot-seek-backwards/91067/2 "2017-07-26T09:03:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.