Fatal exception while booting Elasticsearch

Hi community

My elasticsearch was Up until that i changed all the passwords.

After that i tried starting elasticsearch services and it didn't work.
When i saw the logs in /var/log/elasticsearch/elasticsearch.log
i get it:

[2023-04-26T17:00:57,543][INFO ][o.e.n.Node               ] [DEMO] version[8.7.0], pid[45025], build[deb/09520b59b6bc1057340b55750186466ea715e30e/2023-03-27T16:31:09.816451435Z], OS[Linux/5.10.0-21-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/19.0.2/19.0.2+7-44]
[2023-04-26T17:00:57,551][INFO ][o.e.n.Node               ] [DEMO] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2023-04-26T17:00:57,551][INFO ][o.e.n.Node               ] [DEMO] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=ALL-UNNAMED, -Xms2g, -Xmx2g, -XX:+UseG1GC, -Djava.io.tmpdir=/tmp/elasticsearch-12732300502209698790, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=1073741824, -XX:G1HeapRegionSize=4m, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=15, -Des.distribution.type=deb, --module-path=/usr/share/elasticsearch/lib, --add-modules=jdk.net, -Djdk.module.main=org.elasticsearch.server]
[2023-04-26T17:01:00,491][INFO ][c.a.c.i.j.JacksonVersion ] [DEMO] Package versions: jackson-core=2.13.4, jackson-databind=2.13.4-2, jackson-dataformat-xml=2.13.4, jackson-datatype-jsr310=2.13.4, azure-core=1.34.0, Troubleshooting version conflicts: https://aka.ms/azsdk/java/dependency/troubleshoot
[2023-04-26T17:01:02,744][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [aggregations]
[2023-04-26T17:01:02,744][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [analysis-common]
[2023-04-26T17:01:02,744][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [apm]
[2023-04-26T17:01:02,745][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [blob-cache]
[2023-04-26T17:01:02,745][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [constant-keyword]
[2023-04-26T17:01:02,745][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [data-streams]
[2023-04-26T17:01:02,746][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [frozen-indices]
[2023-04-26T17:01:02,746][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [ingest-attachment]
[2023-04-26T17:01:02,746][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [ingest-common]
[2023-04-26T17:01:02,747][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [ingest-geoip]
[2023-04-26T17:01:02,747][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [ingest-user-agent]
[2023-04-26T17:01:02,747][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [kibana]
[2023-04-26T17:01:02,748][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [lang-expression]
[2023-04-26T17:01:02,748][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [lang-mustache]
[2023-04-26T17:01:02,748][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [lang-painless]
[2023-04-26T17:01:02,749][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [legacy-geo]
[2023-04-26T17:01:02,749][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [mapper-extras]
[2023-04-26T17:01:02,749][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [mapper-version]
[2023-04-26T17:01:02,749][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [old-lucene-versions]
[2023-04-26T17:01:02,750][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [parent-join]
[2023-04-26T17:01:02,750][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [percolator]
[2023-04-26T17:01:02,750][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [rank-eval]
[2023-04-26T17:01:02,751][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [reindex]
[2023-04-26T17:01:02,751][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [repositories-metering-api]
[2023-04-26T17:01:02,751][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [repository-azure]
[2023-04-26T17:01:02,751][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [repository-gcs]
[2023-04-26T17:01:02,752][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [repository-s3]
[2023-04-26T17:01:02,752][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [repository-url]
[2023-04-26T17:01:02,752][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [runtime-fields-common]
[2023-04-26T17:01:02,753][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [search-business-rules]
[2023-04-26T17:01:02,753][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [searchable-snapshots]
[2023-04-26T17:01:02,754][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [snapshot-based-recoveries]
[2023-04-26T17:01:02,754][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [snapshot-repo-test-kit]
[2023-04-26T17:01:02,754][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [spatial]
[2023-04-26T17:01:02,755][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [systemd]
[2023-04-26T17:01:02,755][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [transform]
[2023-04-26T17:01:02,755][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [transport-netty4]
[2023-04-26T17:01:02,756][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [unsigned-long]
[2023-04-26T17:01:02,756][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [vector-tile]
[2023-04-26T17:01:02,756][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [wildcard]
[2023-04-26T17:01:02,757][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-aggregate-metric]
[2023-04-26T17:01:02,757][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-analytics]
[2023-04-26T17:01:02,757][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-async]
[2023-04-26T17:01:02,758][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-async-search]
[2023-04-26T17:01:02,758][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-autoscaling]
[2023-04-26T17:01:02,759][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-ccr]
[2023-04-26T17:01:02,759][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-core]
[2023-04-26T17:01:02,759][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-deprecation]
[2023-04-26T17:01:02,760][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-enrich]
[2023-04-26T17:01:02,760][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-eql]
[2023-04-26T17:01:02,760][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-fleet]
[2023-04-26T17:01:02,760][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-graph]
[2023-04-26T17:01:02,761][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-identity-provider]
[2023-04-26T17:01:02,761][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-ilm]
[2023-04-26T17:01:02,761][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-logstash]
[2023-04-26T17:01:02,762][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-ml]
[2023-04-26T17:01:02,762][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-monitoring]
[2023-04-26T17:01:02,763][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-profiling]
[2023-04-26T17:01:02,763][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-ql]
[2023-04-26T17:01:02,763][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-rollup]
[2023-04-26T17:01:02,764][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-security]
[2023-04-26T17:01:02,764][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-shutdown]
[2023-04-26T17:01:02,764][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-sql]
[2023-04-26T17:01:02,764][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-stack]
[2023-04-26T17:01:02,765][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-text-structure]
[2023-04-26T17:01:02,765][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-voting-only-node]
[2023-04-26T17:01:02,765][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-watcher]
[2023-04-26T17:01:02,766][INFO ][o.e.p.PluginsService     ] [DEMO] loaded module [x-pack-write-load-forecaster]
[2023-04-26T17:01:02,766][INFO ][o.e.p.PluginsService     ] [DEMO] no plugins loaded
[2023-04-26T17:01:07,878][ERROR][o.e.b.Elasticsearch      ] [DEMO] fatal exception while booting Elasticsearch
java.lang.IllegalArgumentException: unknown secure setting [logstash] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:575) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:521) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:491) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:461) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:151) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:56) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:474) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.node.Node.<init>(Node.java:324) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
        at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.7.0.jar:?]
        Suppressed: java.lang.IllegalArgumentException: unknown secure setting [apm_system] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:575) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:521) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:491) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:461) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:151) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:56) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.node.Node.<init>(Node.java:474) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.node.Node.<init>(Node.java:324) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.7.0.jar:?]
        Suppressed: java.lang.IllegalArgumentException: unknown secure setting [kibana_system.password] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:575) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:521) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:491) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:461) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:151) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:56) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.node.Node.<init>(Node.java:474) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.node.Node.<init>(Node.java:324) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.7.0.jar:?]
        Suppressed: java.lang.IllegalArgumentException: unknown secure setting [beats_system] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:575) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:521) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:491) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:461) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:151) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:56) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.node.Node.<init>(Node.java:474) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.node.Node.<init>(Node.java:324) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.7.0.jar:?]
        Suppressed: java.lang.IllegalArgumentException: unknown secure setting [kibana_system] please check that any required plugins are installed, or check the breaking changes documentation for removed settings
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:575) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:521) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:491) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:461) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:151) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:56) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.node.Node.<init>(Node.java:474) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.node.Node.<init>(Node.java:324) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:216) ~[elasticsearch-8.7.0.jar:?]
                at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:67) ~[elasticsearch-8.7.0.jar:?]

¿do you know how can i change the users passwords and enrolled with elasticsearch with theirs new passwords? or ¿is it other problem ?

thanks

The error message seems to say that you created a secure setting named logstash. You should just remove it as it's not a known setting.

Thanks Dadoonet

¿where do i remove that?
As well as the logstash one, I also get the users of kibana, bootstrap

Run this:

bin/elasticsearch-keystore list

Then remove what is not needed. See elasticsearch-keystore | Elasticsearch Guide [8.7] | Elastic

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.