# Elasticsearch index health turned Red! **URL:** \ **Category:** Elasticsearch\ **Created:** [April 27, 2017, 7:18am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805 "2017-04-27T07:18:39Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Abhijit\_Paul](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Abhijit\_Paul](https://discuss.elastic.co/u/Abhijit_Paul)\ **Post date:** [April 27, 2017, 7:18am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/1 "2017-04-27T07:18:39Z")
Hi All, **After index got created in elasticsearch few hours after health of that index turned Red !!!** During my Observation few suspected items are: \> **item1:** Shard Allocation Failed \> **item2:** org.apache.lucene.index. **CorruptIndexException: codec footer mismatch** (file truncated?): actual footer=0 vs expected footer=-1 \> **item3:** hostname hostip - WARN - elasticsearch[master][refresh][T#1] - - - [org.elasticsearch.index.IndexService] [master] [logstash-2017.04.26] `failed to run task refresh - suppressing re-occurring exceptions unless the exception changes` org.elasticsearch.index.engine.RefreshFailedEngineException: Refresh failed at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:658) ~[elasticsearch-5.1.2.jar:5.1.2] ` **Any ideas why it's happening? Is there a guide for recovering from this index health RED to Yellow/Green?** ---
**Author:** ![Abhijit\_Paul](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Abhijit\_Paul](https://discuss.elastic.co/u/Abhijit_Paul)\ **Post date:** [April 27, 2017, 7:19am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/2 "2017-04-27T07:19:27Z")
Above logs ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 27, 2017, 7:40am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/3 "2017-04-27T07:40:27Z")
Hi @Abhijit_Paul, as [I've already written on Github](https://github.com/elastic/elasticsearch/issues/24358#issuecomment-297634137) this is likely due to a problem during the upgrade from Elasticsearch 2.x. Your index files got corrupted as is indicated by messages like `codec footer mismatch (file truncated?): actual footer=0 vs expected footer=-1071082520`. You can see the [upgrade instructions in the docs](https://www.elastic.co/guide/en/elasticsearch/reference/5.3/setup-upgrade.html) and specifically the [upgrade instructions across major versions](https://www.elastic.co/guide/en/elasticsearch/reference/5.3/restart-upgrade.html) (but you should follow the complete upgrade instructions). Daniel ---
**Author:** ![Abhijit\_Paul](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Abhijit\_Paul](https://discuss.elastic.co/u/Abhijit_Paul)\ **Post date:** [April 27, 2017, 8:48am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/4 "2017-04-27T08:48:27Z")
Hi @danielmitterdorfer thanks for your reply. Upgraded Elasticsearch 2.x to 5.x long back, i deployed elk using application descriptor in kubernetes environment, i have two kubernetes stack where both running Elasticsearch 5.1.2, but this Red status only observed only in one of the stack, other stack all fine. That's why i rolling out this upgrade issue, if that is the case then in both the kubernetes stack same behavior should observed. ---
**Author:** ![Abhijit\_Paul](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Abhijit\_Paul](https://discuss.elastic.co/u/Abhijit_Paul)\ **Post date:** [April 27, 2017, 10:20am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/5 "2017-04-27T10:20:51Z")
OK one thing forgot to mentioned that I am using GlusterFS behind the screen, and it's look like there is an issue in GlusterFS **"Elasticsearch get CorruptIndexException errors when running with GlusterFS persistent storage"** due to which index heath turn RED, and this bug is fixed in GlusterFS 3.10 [https://bugzilla.redhat.com/show\_bug.cgi?id=1390050](https://bugzilla.redhat.com/show_bug.cgi?id=1390050) Now i yet to verify the same with GlusterFS 3.10 ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 27, 2017, 10:35am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/6 "2017-04-27T10:35:32Z")
Hi @Abhijit_Paul, thanks for the update. That bug report is indeed an interesting observation. Is there a specific reason why you are running on this file system? Daniel ---
**Author:** ![Abhijit\_Paul](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Abhijit\_Paul](https://discuss.elastic.co/u/Abhijit_Paul)\ **Post date:** [April 27, 2017, 5:07pm UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/7 "2017-04-27T17:07:12Z")
As distributed persistence volume ---
**Author:** ![danielmitterdorfer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmitterdorfer/32/110510_2.png) [@danielmitterdorfer](https://discuss.elastic.co/u/danielmitterdorfer)\ **Post date:** [April 28, 2017, 11:45am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/8 "2017-04-28T11:45:43Z")
Hi @Abhijit_Paul, thanks for the update. I don't know anything about GlusterFS but you should take care using distributed file systems together with Elasticsearch both in terms of performance and data consistency. For example, we also [explicitly advise against using NFS](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/indexing-performance.html#_storage). Daniel ---
**Author:** ![Abhijit\_Paul](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Abhijit\_Paul](https://discuss.elastic.co/u/Abhijit_Paul)\ **Post date:** [April 28, 2017, 5:14pm UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/9 "2017-04-28T17:14:57Z")
yes...true and GusterFS is used as distributed file system in cloud ---
**Author:** ![Yong\_Zhang](https://avatars.discourse-cdn.com/v4/letter/y/a183cd/32.png) [@Yong\_Zhang](https://discuss.elastic.co/u/Yong_Zhang)\ **Post date:** [April 29, 2017, 2:24am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/10 "2017-04-29T02:24:31Z")
I have exactly the same use case and the same issue with you. I'm also running es cluster in Kubernetes and using glusterfs as storage backend, my glusterfs version is 3.10.1, but have the same issue with you, here's my topic [Shard repeat to be UNASSIGNED](https://discuss.elastic.co/t/shard-repeat-to-be-unassigned/83975/3) raised just yesterday... ---
**Author:** ![Abhijit\_Paul](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Abhijit\_Paul](https://discuss.elastic.co/u/Abhijit_Paul)\ **Post date:** [April 29, 2017, 10:42am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/11 "2017-04-29T10:42:10Z")
With GlusterFS version 3.10.0 onward this issue is resolved, found one more issue with combination of GlusterFS & Elasticseach here is the link [https://bugzilla.redhat.com/show\_bug.cgi?id=1430659](https://bugzilla.redhat.com/show_bug.cgi?id=1430659) ---
**Author:** ![Yong\_Zhang](https://avatars.discourse-cdn.com/v4/letter/y/a183cd/32.png) [@Yong\_Zhang](https://discuss.elastic.co/u/Yong_Zhang)\ **Post date:** [May 1, 2017, 9:04am UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/12 "2017-05-01T09:04:45Z")
So the answer is don't run es in Kubernetes? ---
**Author:** ![Abhijit\_Paul](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@Abhijit\_Paul](https://discuss.elastic.co/u/Abhijit_Paul)\ **Post date:** [May 3, 2017, 2:42pm UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/13 "2017-05-03T14:42:40Z")
It's seems with Glusterfs 3.10.1 as well this issue is still persist..... We can use ES with k8s but whether to use along with GlusterFs or not still a mystery ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 31, 2017, 2:44pm UTC](https://discuss.elastic.co/t/elasticsearch-index-health-turned-red/83805/14 "2017-05-31T14:44:47Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.