Likely root cause: org.elasticsearch.gateway.CorruptStateException: codec footer mismatch (file truncated?): actual footer=-993495054 vs expected footer=-1071082520

Elastic Stack Elasticsearch
1

Hello,

I am getting below error when trying to start the ElasticSearch service.
Please let me know if you how to resolve this issue.

[2019-07-26 06:38:15,086][ERROR][bootstrap ] Guice Exception: ElasticsearchException[failed to read [id:16, legacy:false, file:/share/lib/elasticsearch//nodes/1/_state/global-16.st]]; nested: IOException[failed to read [id:16, legacy:false, file:/share/lib/elasticsearch/nodes/1/_state/global-16.st]]; nested: CorruptStateException[codec footer mismatch (file truncated?): actual footer=-993495054 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/share/lib/elasticsearch/afm/nodes/1/_state/global-16.st")))];
Likely root cause: org.elasticsearch.gateway.CorruptStateException: codec footer mismatch (file truncated?): actual footer=-993495054 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/share/lib/elasticsearch/afm/nodes/1/_state/global-16.st")))
at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:418)
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:330)
at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:177)
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:299)
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:119)
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:87)
at org.elasticsearch.gateway.GatewayMetaState.loadMetaState(GatewayMetaState.java:99)
at org.elasticsearch.gateway.GatewayMetaState.pre20Upgrade(GatewayMetaState.java:225)
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:87)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at <<>>
at org.elasticsearch.node.Node.(Node.java:213)
at org.elasticsearch.node.Node.(Node.java:140)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

2

This means that the file /share/lib/elasticsearch//nodes/1/_state/global-16.st is corrupt: it has been changed by something other than Elasticsearch. Unfortunately this file's integrity is vital to Elasticsearch, and I think this means that this node is not recoverable. You will need to start a fresh node with an empty data path and allow Elasticsearch to rebuild any lost shards from the other nodes in the cluster or else restore any lost indices from snapshots.

Also it looks like you're using a really old version of Elasticsearch (2.4 or earlier). It's a good idea to upgrade.

3

Hi David,

We are using ElasticSearch Cluster of three nodes and the data path /share/lib/elasticsearch is shared storage common to all the three nodes.

/share/lib/elasticsearch//nodes/1/_state/global-16.st file is a common file for three nodes and all the nodes are not starting up and giving the same error listed in my previous topic.

4

No, this file is only used by one of the nodes at once. Although all three nodes are using /share/lib/elasticsearch, they subdivide this directory into /share/lib/elasticsearch/nodes/0, /share/lib/elasticsearch/nodes/1 and /share/lib/elasticsearch/nodes/2.

5

Hello David,

I am getting the same error in all the three nodes and but the global-.st files are different.

[2019-07-26 12:38:37,219][ERROR][bootstrap ] Guice Exception: ElasticsearchException[failed to read [id:28, legacy:false, file:/share/lib/elasticsearch/afm/nodes/0/_state/global-28.st]]; nested: IOException[failed to read [id:28, legacy:false, file:/share/lib/elasticsearch/nodes/0/_state/global-28.st]]; nested: CorruptStateException[codec footer mismatch (file truncated?): actual footer=764810365 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/share/lib/elasticsearch/afm/nodes/0/_state/global-28.st")))];
Likely root cause: org.elasticsearch.gateway.CorruptStateException: codec footer mismatch (file truncated?): actual footer=764810365 vs expected footer=-1071082520 (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path="/share/lib/elasticsearch/nodes/0/_state/global-28.st")))
at org.apache.lucene.codecs.CodecUtil.validateFooter(CodecUtil.java:418)
at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:330)
at org.apache.lucene.codecs.CodecUtil.checksumEntireFile(CodecUtil.java:451)
at org.elasticsearch.gateway.MetaDataStateFormat.read(MetaDataStateFormat.java:177)
at org.elasticsearch.gateway.MetaDataStateFormat.loadLatestState(MetaDataStateFormat.java:299)
at org.elasticsearch.gateway.MetaStateService.loadGlobalState(MetaStateService.java:119)
at org.elasticsearch.gateway.MetaStateService.loadFullState(MetaStateService.java:87)
at org.elasticsearch.gateway.GatewayMetaState.loadMetaState(GatewayMetaState.java:99)
at org.elasticsearch.gateway.GatewayMetaState.pre20Upgrade(GatewayMetaState.java:225)
at org.elasticsearch.gateway.GatewayMetaState.(GatewayMetaState.java:87)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at <<>>
at org.elasticsearch.node.Node.(Node.java:213)
at org.elasticsearch.node.Node.(Node.java:140)
at org.elasticsearch.node.NodeBuilder.build(NodeBuilder.java:143)
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:194)
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:286)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:35)

6

In which case something rather serious has happened to your cluster, outside of Elasticsearch's control, and you will need to build a new cluster and restore your data from snapshots.

7

Can we rename global-28.st to global-28.st_bkp and restart the elasticsearch will resolve the issue ?

8

It doesn't sound like a very good idea to start renaming things in the data directory. There's no way of knowing what data might be (silently) lost by doing something like that.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.