ElasticsearchException[failed to bind service]; nested: IndexFormatTooNewException[Format version is not supported

Hello everyone,
I have a issue on my elasticsearch server.
i extended the storage from 1To to 2To and after rebooting i never could restart my elasticsearch service. Here is what i have actually in my journalctl when i try to start it:

Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]: uncaught exception in thread [main]
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]: ElasticsearchException[failed to bind service]; nested: IndexFormatTooNewException[Format version is not supported (resource SimpleFSIndexInput(path="/data/elasticsearch/nodes/0/_state/_2s2.cfs") [slice=_2s2.fdt]): 4 (needs to be between 1 and 3)];
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]: Likely root cause: org.apache.lucene.index.IndexFormatTooNewException: Format version is not supported (resource SimpleFSIndexInput(path="/data/elasticsearch/nodes/0/_state/_2s2.cfs") [slice=_2s2.fdt]): 4 (needs to be between 1 and 3)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.codecs.CodecUtil.checkHeaderNoMagic(CodecUtil.java:216)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:198)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.codecs.CodecUtil.checkIndexHeader(CodecUtil.java:255)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:130)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:123)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.codecs.lucene87.Lucene87StoredFieldsFormat.fieldsReader(Lucene87StoredFieldsFormat.java:131)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:127)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:83)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:66)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:267)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:409)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:330)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.node.Node.<init>(Node.java:362)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.node.Node.<init>(Node.java:289)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.cli.Command.main(Command.java:90)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]:         at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Jan 31 15:59:56 sun-vm-elk systemd-entrypoint[20023]: For complete error details, refer to the log at /var/log/elasticsearch/elasticsearch.log
Jan 31 15:59:56 sun-vm-elk systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan 31 15:59:56 sun-vm-elk systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.
Jan 31 15:59:56 sun-vm-elk systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: https://access.redhat.com/support
-- 
-- Unit elasticsearch.service has failed.
-- 
-- The result is failed.

Any help can be appreciate : )

Welcome to our community! :smiley:

How did you extend the volume?

hello,
After investigation we found that elastic was on 7.17 version and after rebooting the server he downgraded on 7.10 version.
I don't know how that happen but when i tap the yum history command i see that someone performed a downgrading that maybe didnt took effect on 2022 and after that i rebooted the server that took effect and make the elastic service failed.

ID     | Command line                                                                                                                                          | Date and time    | Action(s)      | Altered
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
    18 | update elasticsearch                                                                                                                                  | 2023-01-31 18:48 | Upgrade        |    1 EE
    17 | install --enablerepo=elasticsearch elasticsearch-7.10.2-1                                                                                             | 2022-09-16 16:43 | Downgrade      |    1 EE

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.