Index.lifecycle.rollover_alias [filebeat-7.9.1] does not point to index [filebeat-7.9.1-2020.09.30]

Elastic Stack Elasticsearch
1

Hi everyone!

I was balancing my nodes, and after i get sucessfull i needed to improve my storeage, after i finish this procedure, i got this error in node master:

[ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Could not load codec 'Lucene87'. Did you forget to add lucene-backward-codecs.jar?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.2.jar:7.9.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.2.jar:7.9.2]
Caused by: java.lang.IllegalArgumentException: Could not load codec 'Lucene87'. Did you forget to add lucene-backward-codecs.jar?
at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:449)[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:356) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:262) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:408) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.node.Node.(Node.java:344) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.node.Node.(Node.java:277) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.9.2.jar:7.9.2]
... 6 more
Suppressed: org.apache.lucene.index.CorruptIndexException: checksum passed (4feaaef2). possibly transient resource issue, or a Lucene or JVM bug (resource=BufferedChecksumIndexInput(SimpleFSIndexInput(path=/var/lib/elasticsearcrch/nodes/0/_state/segments_czf")))

            at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:466) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
            at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:434) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
            at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
            at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
            at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
            at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
            at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
            at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
            at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:262) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:408) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:329) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.node.Node.<init>(Node.java:344) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.node.Node.<init>(Node.java:277) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.9.2.jar:7.9.2]
            at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.9.2.jar:7.9.2]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.9.2.jar:7.9.2]
            at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.9.2.jar:7.9.2]

Caused by: java.lang.IllegalArgumentException: An SPI class of type org.apache.lucene.codecs.Codec with name 'Lucene87' does not exist. You need to add the corresponding JAR file supporting this SPI to your classpath. The current clas$
at org.apache.lucene.util.NamedSPILoader.lookup(NamedSPILoader.java:116) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.codecs.Codec.forName(Codec.java:116) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:445) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:356) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) [lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-08-26 10:53:36]
at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:262) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:408) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.node.Node.(Node.java:344) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.node.Node.(Node.java:277) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.9.2.jar:7.9.2]
... 6 more

I already check, i still got the lucene, but seams i have another version, different from what elasticsearch is looking for.
If i rename the file i still geting the same erro.

Please help me out of this?!

2

The title of this topic doesn't match your log file, is that intentional?
Is there more to the log file you can share, before and after this error?

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you :slight_smile:

3

I'm sorry about this mistake, but yes, i have this lucene error.

What i did:

I tried to update elasticsarch thinking about with this update, i could also update de lucene file, but then, i got this new error:

Preformatted textjava.lang.NoSuchMethodError: org.elasticsearch.core.internal.io.IOUtils.closeWhileHandlingException(Ljava/io/Closeable;)V
at org.elasticsearch.common.xcontent.json.JsonXContentParser.close(JsonXContentParser.java:195)
at org.elasticsearch.common.settings.Setting.parseableStringToList(Setting.java:1499)
at org.elasticsearch.common.settings.Setting.lambda$listSetting$44(Setting.java:1478)
at org.elasticsearch.common.settings.Setting.get(Setting.java:441)
at org.elasticsearch.common.settings.Setting.get(Setting.java:435)
at org.elasticsearch.env.Environment.(Environment.java:119)
at org.elasticsearch.env.Environment.(Environment.java:97)
at org.elasticsearch.env.Environment.(Environment.java:93)
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:69)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:91)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:43)
Exception in thread "main" java.lang.NoSuchMethodError: org.elasticsearch.core.internal.io.IOUtils.closeWhileHandlingException(Ljava/io/Closeable;)V
at org.elasticsearch.common.xcontent.json.JsonXContentParser.close(JsonXContentParser.java:195)
at org.elasticsearch.common.settings.Setting.parseableStringToList(Setting.java:1499)
at org.elasticsearch.common.settings.Setting.lambda$listSetting$44(Setting.java:1478)
at org.elasticsearch.common.settings.Setting.get(Setting.java:441)
at org.elasticsearch.common.settings.Setting.get(Setting.java:435)
at org.elasticsearch.env.Environment.(Environment.java:119)
at org.elasticsearch.env.Environment.(Environment.java:97)
at org.elasticsearch.env.Environment.(Environment.java:93)
at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:69)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:100)
at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:91)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127)
at org.elasticsearch.cli.Command.main(Command.java:90)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126)
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)

4

Can you post your Elasticsearch config and more of your log, especially from startup to this error.

6

This is my elasticsearch.yml:

.configuração de rede
network.host: "0.0.0.0"
http.port: 9200
#If set as localhost this will lead this service to be accessed only locally

.Configuração de Cluster
cluster.name: elastic-cluster
node.name: "node1"
node.master: true
node.data: true
node.ingest: true
cluster.initial_master_nodes: ["node1", "node2"]
cluster.max_shards_per_node: 7000

.Path de logs
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: /usr/backup

discovery.zen.ping_timeout: 3s
discovery.zen.join_timeout: 60s
discovery.zen.minimum_master_nodes: 2

discovery.seed_hosts:

  • 10.8.150.xxx:9300
  • 10.8.177.xxx

indices.query.bool.max_clause_count: 8192
search.max_buckets: 250000
xpack.security.enabled: false

7

this is my log file error:

<[2021-01-12T22:08:43,454][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: Could not load codec 'Lucene87'. Did you forget to add lucene-backward-codecs.jar?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.9.2.jar:7.9.2]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.9.2.jar:7.9.2]
Caused by: java.lang.IllegalArgumentException: Could not load codec 'Lucene87'. Did you forget to add lucene-backward-codecs.jar?
at org.apache.lucene.index.SegmentInfos.readCodec(SegmentInfos.java:449) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-$
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:356) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020$
at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020$
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:61) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453$
at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:58) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453$
at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:720) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - iv$
at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:81) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e$
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:63) ~[lucene-core-8.6.2.jar:8.6.2 016993b65e393b58246d54e8ddda9f56a453eb0e - ivera - 2020-$
at org.elasticsearch.gateway.PersistedClusterStateService.nodeMetadata(PersistedClusterStateService.java:262) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:408) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:329) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.node.Node.(Node.java:344) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:227) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.9.2.jar:7.9.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.9.2.jar:7.9.2]
... 6 more

Seems like a mismatch of lucene version, caused by node2, cause node1 has elasticsearch 7.9.x and node 2 as elasticsearch version 7.1.0, and few weeks ago, I had balanced the nodes, maybe there is a record which was in some directory that cause this mismatch. This is what I think is the possible cause for this error.

8

That's really not a good idea and will lead to a number of issues.

So will this.

And why is this so high?

You should upgrade all your nodes to be on the same version ASAP.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.