# Elasticsearch shard unassigned and changed to red
**URL:** https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729
**Category:** Elasticsearch
**Created:** [June 28, 2018, 5:01am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729 "2018-06-28T05:01:48Z")
**Posts on this page:** 17
**Page:** 1
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 28, 2018, 5:01am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/1 "2018-06-28T05:01:48Z")
Hi
I am using logstash to ship my log data to es, the index is created in the format log-YYYY-mm-dd. It was running fine for 4-5 hours and then the shards failed, turned red, in log its written Engine failed[merge failed]... Corrupted index... and all
What is the real issue?
Even after failing I am able to search the es for few more hours. But cant write data into it at all. After few more hours the search also fails.
The log file is attached with the mail
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 28, 2018, 7:16am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/2 "2018-06-28T07:16:31Z")
Which version of Elasticsearch are you using? What is the specification of your cluster? What kind of storage are you using?
---
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 28, 2018, 10:01am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/3 "2018-06-28T10:01:42Z")
version 6 (latest)
Everything is default, i just reduced memory in jvm.options to 600mb, Kind of storage means?
I am running elasticsearch inside a docker container, logstash also runs inside the same container
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 28, 2018, 10:06am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/4 "2018-06-28T10:06:28Z")
> [@devika\_sudheer](#):
>
> Kind of storage means?
Are you using local SSD or spinning disks, SAN, NFS or some other type of storage?
---
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 29, 2018, 7:28am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/5 "2018-06-29T07:28:42Z")
Actually that is unknown as elasticsearch run on a remote server. The storage details of that host is unknown. But how does that affect es?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 29, 2018, 7:35am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/6 "2018-06-29T07:35:10Z")
That is a very small heap for running Elasticsearch, but should not cause corruption. The reason I am asking about the storage is that some types of storage do not work with Elasticsearch, and can cause corruption. See these posts for an example:
> [@Underlying file changed by an external force](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/137555/4):
>
> The first two Google hits for elasticsearch glusterfs lead to two serious-looking open GlusterFS issues that Elasticsearch triggers: [https://bugzilla.redhat.com/show\_bug.cgi?id=1390050](https://bugzilla.redhat.com/show_bug.cgi?id=1390050)[https://bugzilla.redhat.com/show\_bug.cgi?id=1379568](https://bugzilla.redhat.com/show_bug.cgi?id=1379568) This may be one of them, or may be something else, but it seems worth rethinking your setup. I don't think so. Both of the times I quoted are in UTC (note the trailing Z) and differ by 28 seconds, so it seems unlikely to be a timezone issue.
> [@Elk on Docker Swarm and glusterFS crash](https://discuss.elastic.co/t/elk-on-docker-swarm-and-glusterfs-crash/127940):
>
> Hi, I'm trying to deploy an ELK stack on docker swarm. If I bind the elastic dir data to a Docker volume there is no problem. The problems comes as soon as I try to bind the elstastic data dir to a glusterFS volume. I use glusterFS to synchronise the data between all the swarm nodes in the cluster. I deploy ELK using the following code: elasticsearch: image: docker.elastic.co/elasticsearch/elasticsearch:6.2.3 # container\_name: elasticsearch environment: - "http.host=0.0.…
---
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 29, 2018, 7:51am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/7 "2018-06-29T07:51:40Z")
There is already another app which uses elasticsearch running successfully in the same host so the storage type is not a issue
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 29, 2018, 7:54am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/8 "2018-06-29T07:54:48Z")
Is that use case that app covering similar to yours? What is the full error message you are seeing?
---
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 29, 2018, 10:07am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/9 "2018-06-29T10:07:42Z")
Yes
I can't search elasticsearch.
It is red.
Request fails, that is the error
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 29, 2018, 10:10am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/10 "2018-06-29T10:10:50Z")
What is in the Elasticsearch logs?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [June 29, 2018, 10:37am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/11 "2018-06-29T10:37:04Z")
> [@devika\_sudheer](#):
>
> The log file is attached with the mail
Attachments are not processed in this forum. So you need to upload that to a 3rd party service like [gist.github.com](http://gist.github.com) for example and add the link in your post.
---
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 30, 2018, 5:37am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/12 "2018-06-30T05:37:49Z")
[2018-06-21T02:50:22,523][INFO][o.e.c.r.a.AllocationService] [RjaKnLJ] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[ptp-2018-06-21][2]] ...]).
[2018-06-21T04:29:54,733][WARN][o.e.i.e.Engine] [RjaKnLJ] [ptp-2018-06-21][2] failed engine [merge failed]
org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=10 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/app/data/elasticsearch/nodes/0/indices/wN2Gi9erR\_uh4kLASV4SmA/2/index/\_13m\_Lucene70\_0.dvm")))
```
at org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2111) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) [elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
```
Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=10 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/app/data/elasticsearch/nodes/0/indices/wN2Gi9erR\_uh4kLASV4SmA/2/index/\_13m\_Lucene70\_0.dvm")))
```
at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:502) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:103) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5010) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4507) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
```
[2018-06-21T04:29:54,848][WARN][o.e.i.c.IndicesClusterStateService] [RjaKnLJ] [[ptp-2018-06-21][2]] marking and sending shard failed due to [shard failure, reason [merge failed]]
org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=10 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/app/data/elasticsearch/nodes/0/indices/wN2Gi9erR\_uh4kLASV4SmA/2/index/\_13m\_Lucene70\_0.dvm")))
```
at org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2111) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
```
Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=10 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/app/data/elasticsearch/nodes/0/indices/wN2Gi9erR\_uh4kLASV4SmA/2/index/\_13m\_Lucene70\_0.dvm")))
```
at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:502) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.codecs.lucene50.Lucene50CompoundFormat.write(Lucene50CompoundFormat.java:103) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.createCompoundFile(IndexWriter.java:5010) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4507) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
```
---
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 30, 2018, 5:38am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/13 "2018-06-30T05:38:00Z")
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:4083) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
```
at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:624) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
at org.elasticsearch.index.engine.ElasticsearchConcurrentMergeScheduler.doMerge(ElasticsearchConcurrentMergeScheduler.java:99) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:661) ~[lucene-core-7.2.1.jar:7.2.1 b2b6438b37073bee1fca40374e85bf91aa457c0b - ubuntu - 2018-01-10 00:48:43]
```
[2018-06-21T04:29:54,852][WARN][o.e.c.a.s.ShardStateAction] [RjaKnLJ] [ptp-2018-06-21][2] received shard failed for shard id [[ptp-2018-06-21][2]], allocation id [omffE7HcSo25TTYLNbZuTA], primary term [0], message [shard failure, reason [merge failed]], failure [MergeException[org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=10 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/app/data/elasticsearch/nodes/0/indices/wN2Gi9erR\_uh4kLASV4SmA/2/index/\_13m\_Lucene70\_0.dvm")))]; nested: CorruptIndexException[codec footer mismatch (file truncated?): actual footer=10 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/app/data/elasticsearch/nodes/0/indices/wN2Gi9erR\_uh4kLASV4SmA/2/index/\_13m\_Lucene70\_0.dvm")))]; ]
org.apache.lucene.index.MergePolicy$MergeException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=10 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/app/data/elasticsearch/nodes/0/indices/wN2Gi9erR\_uh4kLASV4SmA/2/index/\_13m\_Lucene70\_0.dvm")))
```
at org.elasticsearch.index.engine.InternalEngine$EngineMergeScheduler$2.doRun(InternalEngine.java:2111) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:672) ~[elasticsearch-6.2.4.jar:6.2.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.2.4.jar:6.2.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_171]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_171]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
```
---
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 30, 2018, 5:38am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/14 "2018-06-30T05:38:27Z")
These are the logs
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 30, 2018, 5:46am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/15 "2018-06-30T05:46:53Z")
I am still interested to know what the underlying storage is. Is the other use case you mentioned also a write-intensive log analytics use case with similar characteristics?
---
### Author:  [@devika\_sudheer](https://discuss.elastic.co/u/devika_sudheer)
#### Post date: [June 30, 2018, 5:58am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/16 "2018-06-30T05:58:44Z")
Yes
Both are similar applications
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 28, 2018, 6:10am UTC](https://discuss.elastic.co/t/elasticsearch-shard-unassigned-and-changed-to-red/137729/17 "2018-07-28T06:10:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.