# Elasticsearch on Kubernetes - cluster red, one node cannot start **URL:** \ **Category:** Elasticsearch\ **Created:** [December 21, 2019, 2:03pm UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-cluster-red-one-node-cannot-start/212738 "2019-12-21T14:03:05Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![dmge](https://avatars.discourse-cdn.com/v4/letter/d/b782af/32.png) [@dmge](https://discuss.elastic.co/u/dmge)\ **Post date:** [December 21, 2019, 2:03pm UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-cluster-red-one-node-cannot-start/212738/1 "2019-12-21T14:03:05Z")
Hello, Could you please help me to take back my cluster and shards green? I am using Elastisearch on Kubernetes pods. Have cluster with 3 nodes. After some restarts of the pods/nodes the 3th node does not start, cluster is red, 2 important shards are not assigned. I have the following types of logs in this node (failed to read node state and .index.CorruptIndexException: codec footer mismatch): ``` [2019-12-21T11:33:23,329][ERROR][o.e.g.GatewayMetaState] [elasticsearch-2] failed to read local state, exiting... org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:80, file:/usr/share/elasticsearch/data/nodes/0/_state/global-80.st] [2019-12-21T11:33:23,329][ERROR][o.e.g.GatewayMetaState] [elasticsearch-2] failed to read local state, exiting... org.elasticsearch.ElasticsearchException: java.io.IOException: failed to read [id:80, file:/usr/share/elasticsearch/data/nodes/0/_state/global-80.st] at org.elasticsearch.ExceptionsHelper.maybeThrowRuntimeAndSuppress(ExceptionsHelper.java:165) ~[elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:306) ~[elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:324) ~[elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:112) ~[elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:57) ~[elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:88) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.node.Node.(Node.java:499) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.node.Node.(Node.java:266) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:212) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:333) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) [elasticsearch-cli-6.8.2.jar:6.8.2] at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-6.8.2.jar:6.8.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116) [elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93) [elasticsearch-6.8.2.jar:6.8.2] Caused by: java.io.IOException: failed to read [id:80, file:/usr/share/elasticsearch/data/nodes/0/_state/global-80.st] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:300) ~[elasticsearch-6.8.2.jar:6.8.2] ... 16 more Caused by: org.elasticsearch.gateway.CorruptStateException: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=-470888448 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/global-80.st"))) at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:202) ~[elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:296) ~[elasticsearch-6.8.2.jar:6.8.2] ... 16 more Caused by: org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=-470888448 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/global-80.st"))) at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:502) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28] at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:414) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28] at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:526) ~[lucene-core-7.7.0.jar:7.7.0 8c831daf4eb41153c25ddb152501ab5bae3ea3d5 - jimczi - 2019-02-04 23:16:28] at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:185) ~[elasticsearch-6.8.2.jar:6.8.2] at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestStateWithGeneration(MetaDataStateFormat.java:296) ~[elasticsearch-6.8.2.jar:6.8.2] ... 16 more [2019-12-21T11:33:23,390][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-2] uncaught exception in thread [main] [2019-12-21T11:33:23,390][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [elasticsearch-2] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: ElasticsearchException[java.io.IOException: failed to read [id:80, file:/usr/share/elasticsearch/data/nodes/0/_state/global-80.st]]; nested: IOException[failed to read [id:80, file:/usr/share/elasticsearch/data/nodes/0/_state/global-80.st]]; nested: CorruptStateException[org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=-470888448 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/global-80.st")))]; nested: CorruptIndexException[codec footer mismatch (file truncated?): actual footer=-470888448 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/global-80.st")))]; ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 22, 2019, 9:55am UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-cluster-red-one-node-cannot-start/212738/2 "2019-12-22T09:55:18Z")
The lack of line breaks make this pretty hard to read, but this is the fundamental problem: `org.apache.lucene.index.CorruptIndexException: codec footer mismatch (file truncated?): actual footer=-470888448 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/usr/share/elasticsearch/data/nodes/0/_state/global-80.st")))` The metadata on disk is corrupt, or at least it is different from the metadata that Elasticsearch wrote. This indicates something wrong with your storage or filesystem. Elasticsearch presents a workload to the underlying disks that's quite good at exposing bugs (e.g. [in the kernel](https://bugs.launchpad.net/ubuntu/+source/linux-azure/+bug/1848739/comments/9) or [in the filesystem](https://discuss.elastic.co/t/underlying-file-changed-by-an-external-force/137555/4)) as well as faulty hardware. After addressing that, I think it's best to start this node afresh by wiping the data directory and restore any red indices from a recent snapshot. ---
**Author:** ![dmge](https://avatars.discourse-cdn.com/v4/letter/d/b782af/32.png) [@dmge](https://discuss.elastic.co/u/dmge)\ **Post date:** [December 22, 2019, 10:29am UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-cluster-red-one-node-cannot-start/212738/3 "2019-12-22T10:29:31Z")
Hi David, Thank you very much for your answer. I highly appreciate it! Could you please share your opinion and help me clarify a few more things: Because I do not have snapshot of the data and I do not have understanding of the Elasticsearch logic behind the data management, I am trying to figure out our best options: 1. If we try to resolve the issues of the underlying storage volume (yes, it is GlusterFS), is there a chance the data from these 2 shards to be joined back to the Elasticsearch data? What approach we can use to join back the data? 2. If we move the data directory on other location, after the cluster is up again, is there a option to to restore the 2 shards from the moved old data? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 22, 2019, 12:02pm UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-cluster-red-one-node-cannot-start/212738/4 "2019-12-22T12:02:19Z")
Unfortunately the only other way forward is to create the index from the source data again. If you no longer have the source data then I don't have any recovery suggestions. Elasticsearch works pretty hard to protect the integrity of your data, but of course there's a limit to how resilient it is reasonable to be. Your daring choice of filesystem plus the lack of replicas or snapshots is just too much to handle. ---
**Author:** ![dmge](https://avatars.discourse-cdn.com/v4/letter/d/b782af/32.png) [@dmge](https://discuss.elastic.co/u/dmge)\ **Post date:** [December 22, 2019, 7:42pm UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-cluster-red-one-node-cannot-start/212738/5 "2019-12-22T19:42:13Z")
> [@DavidTurner](#): > > Unfortunately the only other way forward is to create the index from the source data again. Hi David, Do you mean the whole index? Could I use the other 3 healthy shards of this index? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 22, 2019, 10:55pm UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-cluster-red-one-node-cannot-start/212738/6 "2019-12-22T22:55:37Z")
You can, but I don't see how it helps since each shard contains some random fraction of the documents in an index. The reference manual contains instructions for accepting this kind of data loss by [allocating a stale or empty primary](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/cluster-reroute.html#cluster-reroute). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 19, 2020, 10:55pm UTC](https://discuss.elastic.co/t/elasticsearch-on-kubernetes-cluster-red-one-node-cannot-start/212738/7 "2020-01-19T22:55:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.