# A shard closed suddenly on indexing data
**URL:** https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170
**Category:** Elasticsearch
**Created:** [August 7, 2019, 7:28am UTC](https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170 "2019-08-07T07:28:56Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@chenchuangc](https://discuss.elastic.co/u/chenchuangc)
#### Post date: [August 7, 2019, 7:28am UTC](https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170/1 "2019-08-07T07:28:56Z")
**Elasticsearch version**
```auto
bin/elasticsearch --version
7.1.1
```
**Logstash version**
```auto
7.1.1
```
**Plugins installed**
```auto
bin/elasticsearch-plugin list
analysis-ik
```
**JVM version**
```auto
java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)
```
**OS version**
```auto
uname -a
Linux log-es05.com 2.6.32-642.6.2.el6.x86_64 #1 SMP Wed Oct 26 06:52:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
```
**Related config**
```auto
node.master: false
node.data: true
node.ingest: true
node.ml: false
xpack.ml.enabled: true
```
## issue
i am doing log analysis using Filebeat (7.1.1) -\> logstash(7.1.1) -\> Elasticsearch (7.1.1)
but there is a sudden relocation of my index, and i find some error log from the master node
if you want the total , i also give the total file [here](if you want the total , i also give the total file here [https://github.com/chenchuangc/test\_git/blob/master/es\_error.md](https://github.com/chenchuangc/test_git/blob/master/es_error.md))
`https://github.com/chenchuangc/test_git/blob/master/es_error.md`
```auto
[2019-08-07T14:52:05,693][WARN][o.e.c.r.a.AllocationService] [ES01] failing shard [failed shard, shard [ktest-service-2019.08.07][24], node[fkr_Uo8IROeC-SN2Qs0MbA], [R], s[STARTED], a[id=ViYqHgZBR4mdVXa76ZDyUQ], message [failed to perform indices:data/write/bulk[s] on replica [ktest-service-2019.08.07][24], node[fkr_Uo8IROeC-SN2Qs0MbA], [R], s[STARTED], a[id=ViYqHgZBR4mdVXa76ZDyUQ]], failure [RemoteTransportException[[ESV11][10.66.3.247:9300][indices:data/write/bulk[s][r]]]; nested: AlreadyClosedException[[ktest-service-2019.08.07][24] engine is closed]; nested: ArrayIndexOutOfBoundsException[67]; ], markAsStale [true]]
org.elasticsearch.transport.RemoteTransportException: [ESV11][10.66.3.247:9300][indices:data/write/bulk[s][r]]
Caused by: org.apache.lucene.store.AlreadyClosedException: [ktest-service-2019.08.07][24] engine is closed
at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:789) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.engine.Engine.ensureOpen(Engine.java:798) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:826) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:789) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:762) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnReplica(IndexShard.java:726) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOpOnReplica(TransportShardBulkAction.java:416) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnReplica(TransportShardBulkAction.java:386) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:373) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.bulk.TransportShardBulkAction.shardOperationOnReplica(TransportShardBulkAction.java:79) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:628) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.onResponse(TransportReplicationAction.java:588) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2733) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard$4.onResponse(IndexShard.java:2711) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:269) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShardOperationPermits.acquire(IndexShardOperationPermits.java:236) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard.lambda$acquireReplicaOperationPermit$18(IndexShard.java:2671) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard.innerAcquireReplicaOperationPermit(IndexShard.java:2778) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard.acquireReplicaOperationPermit(IndexShard.java:2670) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction.acquireReplicaOperationPermit(TransportReplicationAction.java:992) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$AsyncReplicaAction.doRun(TransportReplicationAction.java:698) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:571) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicaOperationTransportHandler.messageReceived(TransportReplicationAction.java:556) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:300) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:341) ~[?:?]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ListenableFuture$1.run(ListenableFuture.java:99) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:192) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:92) ~[elasticsearch-7.1.1.jar:7.1.1]
at
.....
```
---
### Author:  [@chenchuangc](https://discuss.elastic.co/u/chenchuangc)
#### Post date: [August 7, 2019, 7:30am UTC](https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170/2 "2019-08-07T07:30:15Z")
and there is also a cause by , to long to be at the whole post
```auto
Caused by: java.lang.ArrayIndexOutOfBoundsException: 67
at org.apache.lucene.codecs.lucene50.ForUtil.readBlock(ForUtil.java:196) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockPostingsEnum.refillDocs(Lucene50PostingsReader.java:615) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.codecs.lucene50.Lucene50PostingsReader$BlockPostingsEnum.nextDoc(Lucene50PostingsReader.java:661) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.index.MappingMultiPostingsEnum$MappingPostingsSub.nextDoc(MappingMultiPostingsEnum.java:51) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.index.DocIDMerger$SequentialDocIDMerger.next(DocIDMerger.java:99) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.index.MappingMultiPostingsEnum.nextDoc(MappingMultiPostingsEnum.java:103) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.codecs.PushPostingsWriterBase.writeTerm(PushPostingsWriterBase.java:135) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter$TermsWriter.write(BlockTreeTermsWriter.java:865) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.codecs.blocktree.BlockTreeTermsWriter.write(BlockTreeTermsWriter.java:344) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.codecs.FieldsConsumer.merge(FieldsConsumer.java:105) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsWriter.merge(PerFieldPostingsFormat.java:169) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.index.SegmentMerger.mergeTerms(SegmentMerger.java:244) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:139) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4459) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4054) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:625) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:101) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:662) ~[lucene-core-8.0.0.jar:8.0.0 2ae4746365c1ee72a0047ced7610b2096e438979 - jimczi - 2019-03-08 11:58:55]
[2019-08-07T14:52:05,800][INFO][o.e.c.r.a.AllocationService] [ES01] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[ktest-service-2019.08.07][24]] ...]).
```
---
### Author:  [@AlanWoodward](https://discuss.elastic.co/u/AlanWoodward)
#### Post date: [August 7, 2019, 8:52am UTC](https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170/3 "2019-08-07T08:52:38Z")
> [@chenchuangc](#):
>
> at org.apache.lucene.codecs.lucene50.ForUtil.readBlock
This looks like index corruption to me - there's a similar Lucene issue at [[LUCENE-8252] ArrayIndexOutOfBoundsException hit in lucene50.ForUtil.readBlock - ASF JIRA](https://issues.apache.org/jira/browse/LUCENE-8252). Can you run checkindex on the problematic shard? See [A Dive into the Elasticsearch Storage | Elastic Blog](https://www.elastic.co/blog/found-dive-into-elasticsearch-storage#fixing-problematic-shards) for details on how to run it.
---
### Author:  [@chenchuangc](https://discuss.elastic.co/u/chenchuangc)
#### Post date: [August 8, 2019, 2:48am UTC](https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170/4 "2019-08-08T02:48:57Z")
thank you very much @AlanWoodward
1. After the error the cluster do a reroute (using 5min for 10G data) automatic, and there is no problematic shard in the origin node. so i can not `run checkindex on the problematic shard`
2. you see, the lucene issue affect version is 6.6.2
but i see my es cluster is
```auto
{
"name" : "ESV09",
"cluster_name" : "log-application",
"cluster_uuid" : "i_y2nBUxRrWLydBgAlzilg",
"version" : {
"number" : "7.1.1",
"build_flavor" : "default",
"build_type" : "tar",
"build_hash" : "7a013de",
"build_date" : "2019-05-23T14:04:00.380842Z",
"build_snapshot" : false,
"lucene_version" : "8.0.0",
"minimum_wire_compatibility_version" : "6.8.0",
"minimum_index_compatibility_version" : "6.0.0-beta1"
},
"tagline" : "You Know, for Search"
}
```
lucene\_version is 8.0.0
---
### Author:  [@AlanWoodward](https://discuss.elastic.co/u/AlanWoodward)
#### Post date: [August 9, 2019, 8:09am UTC](https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170/5 "2019-08-09T08:09:22Z")
I don't think it's due to a bug in Lucene, but rather a hardware or OS/network error writing data. Has it happened again? If it's a one-off then this is just elasticsearch detecting a problem with the data and repairing it, but if it's recurrent then we'll need to dig deeper.
---
### Author:  [@chenchuangc](https://discuss.elastic.co/u/chenchuangc)
#### Post date: [August 9, 2019, 9:47am UTC](https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170/6 "2019-08-09T09:47:13Z")
thank you so much , i hava just met this one time from the cluster start , maybe just like you said ,it is a hardware error, i will continue to watch about it .
thank you one more time 😁
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 6, 2019, 9:47am UTC](https://discuss.elastic.co/t/a-shard-closed-suddenly-on-indexing-data/194170/7 "2019-09-06T09:47:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.