Logstash fails after upgrading to version - 8.15.1

Elastic Stack Logstash
,
1

Hi Team, Today i upgraded my Elasticsearch, Logstash, and Kibana stack from version 8.6.2 to 8.15.1. My setup is based on Docker Compose.

After the upgrade, the Logstash container fails to start and throws the following error:

ERROR: Failed to load settings file from "path.settings". Aborting...
path.settings=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate ${xpack.monitoring.elasticsearch.password}. Replacement variable xpack.monitoring.elasticsearch.password is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
[FATAL] 2024-09-10 10:43:39.349 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit

It seems that Logstash cannot find or evaluate the variable ${xpack.monitoring.elasticsearch.password} which was working fine before the upgrade. I’ve checked my environment variables and secret store, but I’m not sure what could be causing this issue after the version change.

Could anyone advise on what might be causing this error or how to resolve it?

Thank you in advance!

1 Like
2

Do you mean 8.15.1? I do not see any indication 8.15.2 is out yet.

8.15.1 includes a tweak to the way ${} is evaluated and it looks like there is already an open issue for it. Does downgrading to 8.15.0 fix things?

3

Hi @Badger - Sorry for the inconvenience!! yes it is 8.15.1 version.

I haven't downgraded yet, So i will check the same and get back.

4

@Badger - I have downgraded my Elasticsearch, Logstash, and Kibana stack version to 8.15.0, but I'm still getting the same error in my Logstash container, The logs as follows.

logstash  | ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate `${xpack.monitoring.elasticsearch.password}`. Replacement variable `xpack.monitoring.elasticsearch.password` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
logstash  | [FATAL] 2024-09-11 05:12:22.470 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
logstash  | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
logstash  |     at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
logstash  | 2024/09/11 05:12:35 Setting 'xpack.monitoring.elasticsearch.password' from environment.
logstash  | 2024/09/11 05:12:35 Setting 'xpack.monitoring.enabled' from environment.
logstash  | 2024/09/11 05:12:35 Setting 'xpack.monitoring.elasticsearch.username' from environment.
logstash  | Using bundled JDK: /usr/share/logstash/jdk

Whereas in Elasticsearch, it says cannot downgrade a node from version [8.15.1] to version [8.15.0]. Here is the complete error message.

elasticsearch  | {"@timestamp":"2024-09-11T05:12:41.267Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster","error.type":"java.lang.IllegalStateException","error.message":"cannot downgrade a node from version [8.15.1] to version [8.15.0]","error.stack_trace":"java.lang.IllegalStateException: cannot downgrade a node from version [8.15.1] to version [8.15.0]\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.env.NodeMetadata.verifyUpgradeToCurrentVersion(NodeMetadata.java:134)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:140)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:637)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:334)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:513)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:260)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.node.Node.<init>(Node.java:192)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:242)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:242)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:76)\n"}
elasticsearch  | ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log
elasticsearch  |
elasticsearch  | ERROR: Elasticsearch died while starting up, with exit code 1
5

Hi @mohanss08 ,

Recently , I also upgraded my logstash to 8.15.1 version and am facing the same error as you.

"LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: Cannot evaluate ${BOOTSTRAP_SERVERS}. Replacement variable BOOTSTRAP_SERVERS is not defined in a Logstash secret store or an environment entry and there is no default value given.","

Were you able to get any success on this issue?

Regards
Nalin

6

Hi Nalin, The problem still i couldn't able to fix it. May be we need try with next version - 8.15.2

7

Have you downgraded Logstash?

You cannot downgrade Elasticsearch and Kibana, but you can downgrade Logstash, try to downgrade it to 8.14.3.

8

@leandrojmp

Same problem occurring even with logstash v-8.14.3 & 8.13.4 as well.

logstash  | ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate `${xpack.monitoring.elasticsearch.password}`. Replacement variable `xpack.monitoring.elasticsearch.password` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
logstash  | [FATAL] 2024-09-19 03:30:03.596 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
logstash  | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
logstash  |     at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
logstash  | 2024/09/19 03:30:04 Setting 'xpack.monitoring.elasticsearch.password' from environment.
logstash  | 2024/09/19 03:30:04 Setting 'xpack.monitoring.enabled' from environment.
logstash  | 2024/09/19 03:30:04 Setting 'xpack.monitoring.elasticsearch.username' from environment.
logstash  | Using bundled JDK: /usr/share/logstash/jdk
9

Yeah, just checked the issue linked, the bug was introduced on 8.13.1 as you can check here.

After 8.13.1, running Logstash on docker with list environment variable (example: -e XPACK_MANAGEMENT_ELASTICSEARCH_HOSTS='[es.host.1, es.host.2:9500]' ) doesn't work.

So you need to try with a version before 8.13.1 like 8.13.0 or 8.12.2, you can also still use Logstash 8.6.2 where it was working.

10

@leandrojmp - Thanks it works with logstash-8.12.2 version.

11

Thanks for this topic. I was about to upgrade to 8.15. I will wait for new version