Logstash fails after upgrading to version - 8.15.1

Elastic Stack Logstash
,
1

Hi Team, Today i upgraded my Elasticsearch, Logstash, and Kibana stack from version 8.6.2 to 8.15.1. My setup is based on Docker Compose.

After the upgrade, the Logstash container fails to start and throws the following error:

ERROR: Failed to load settings file from "path.settings". Aborting...
path.settings=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate ${xpack.monitoring.elasticsearch.password}. Replacement variable xpack.monitoring.elasticsearch.password is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
[FATAL] 2024-09-10 10:43:39.349 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit

It seems that Logstash cannot find or evaluate the variable ${xpack.monitoring.elasticsearch.password} which was working fine before the upgrade. I’ve checked my environment variables and secret store, but I’m not sure what could be causing this issue after the version change.

Could anyone advise on what might be causing this error or how to resolve it?

Thank you in advance!

1 Like
2

Do you mean 8.15.1? I do not see any indication 8.15.2 is out yet.

8.15.1 includes a tweak to the way ${} is evaluated and it looks like there is already an open issue for it. Does downgrading to 8.15.0 fix things?

3

Hi @Badger - Sorry for the inconvenience!! yes it is 8.15.1 version.

I haven't downgraded yet, So i will check the same and get back.

4

@Badger - I have downgraded my Elasticsearch, Logstash, and Kibana stack version to 8.15.0, but I'm still getting the same error in my Logstash container, The logs as follows.

logstash  | ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate `${xpack.monitoring.elasticsearch.password}`. Replacement variable `xpack.monitoring.elasticsearch.password` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
logstash  | [FATAL] 2024-09-11 05:12:22.470 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
logstash  | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
logstash  |     at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
logstash  | 2024/09/11 05:12:35 Setting 'xpack.monitoring.elasticsearch.password' from environment.
logstash  | 2024/09/11 05:12:35 Setting 'xpack.monitoring.enabled' from environment.
logstash  | 2024/09/11 05:12:35 Setting 'xpack.monitoring.elasticsearch.username' from environment.
logstash  | Using bundled JDK: /usr/share/logstash/jdk

Whereas in Elasticsearch, it says cannot downgrade a node from version [8.15.1] to version [8.15.0]. Here is the complete error message.

elasticsearch  | {"@timestamp":"2024-09-11T05:12:41.267Z", "log.level":"ERROR", "message":"fatal exception while booting Elasticsearch", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"main","log.logger":"org.elasticsearch.bootstrap.Elasticsearch","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"docker-cluster","error.type":"java.lang.IllegalStateException","error.message":"cannot downgrade a node from version [8.15.1] to version [8.15.0]","error.stack_trace":"java.lang.IllegalStateException: cannot downgrade a node from version [8.15.1] to version [8.15.0]\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.env.NodeMetadata.verifyUpgradeToCurrentVersion(NodeMetadata.java:134)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.env.NodeMetadata.upgradeToCurrentVersion(NodeMetadata.java:140)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.env.NodeEnvironment.loadNodeMetadata(NodeEnvironment.java:637)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:334)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.node.NodeConstruction.validateSettings(NodeConstruction.java:513)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.node.NodeConstruction.prepareConstruction(NodeConstruction.java:260)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.node.Node.<init>(Node.java:192)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.bootstrap.Elasticsearch$2.<init>(Elasticsearch.java:242)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.bootstrap.Elasticsearch.initPhase3(Elasticsearch.java:242)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:76)\n"}
elasticsearch  | ERROR: Elasticsearch did not exit normally - check the logs at /usr/share/elasticsearch/logs/docker-cluster.log
elasticsearch  |
elasticsearch  | ERROR: Elasticsearch died while starting up, with exit code 1
5

Hi @mohanss08 ,

Recently , I also upgraded my logstash to 8.15.1 version and am facing the same error as you.

"LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: Cannot evaluate ${BOOTSTRAP_SERVERS}. Replacement variable BOOTSTRAP_SERVERS is not defined in a Logstash secret store or an environment entry and there is no default value given.","

Were you able to get any success on this issue?

Regards
Nalin

6

Hi Nalin, The problem still i couldn't able to fix it. May be we need try with next version - 8.15.2

7

Have you downgraded Logstash?

You cannot downgrade Elasticsearch and Kibana, but you can downgrade Logstash, try to downgrade it to 8.14.3.

8

@leandrojmp

Same problem occurring even with logstash v-8.14.3 & 8.13.4 as well.

logstash  | ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate `${xpack.monitoring.elasticsearch.password}`. Replacement variable `xpack.monitoring.elasticsearch.password` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
logstash  | [FATAL] 2024-09-19 03:30:03.596 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
logstash  | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
logstash  |     at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
logstash  | 2024/09/19 03:30:04 Setting 'xpack.monitoring.elasticsearch.password' from environment.
logstash  | 2024/09/19 03:30:04 Setting 'xpack.monitoring.enabled' from environment.
logstash  | 2024/09/19 03:30:04 Setting 'xpack.monitoring.elasticsearch.username' from environment.
logstash  | Using bundled JDK: /usr/share/logstash/jdk
9

Yeah, just checked the issue linked, the bug was introduced on 8.13.1 as you can check here.

After 8.13.1, running Logstash on docker with list environment variable (example: -e XPACK_MANAGEMENT_ELASTICSEARCH_HOSTS='[es.host.1, es.host.2:9500]' ) doesn't work.

So you need to try with a version before 8.13.1 like 8.13.0 or 8.12.2, you can also still use Logstash 8.6.2 where it was working.

10

@leandrojmp - Thanks it works with logstash-8.12.2 version.

11

Thanks for this topic. I was about to upgrade to 8.15. I will wait for new version

12

Hi All,

Today tried with Logstash - 8.15.2 version, however still same problem.

logstash  | ERROR: Failed to load settings file from "path.settings". Aborting... path.setting=/usr/share/logstash/config, exception=LogStash::ConfigurationError, message=>Cannot evaluate `${xpack.monitoring.elasticsearch.password}`. Replacement variable `xpack.monitoring.elasticsearch.password` is not defined in a Logstash secret store or as an Environment entry and there is no default value given.
logstash  | [FATAL] 2024-10-09 05:30:54.028 [main] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
logstash  | org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
logstash  |     at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
logstash  |     at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
logstash  | 2024/10/09 05:30:54 Setting 'xpack.monitoring.elasticsearch.password' from environment.
logstash  | 2024/10/09 05:30:54 Setting 'xpack.monitoring.enabled' from environment.
logstash  | 2024/10/09 05:30:54 Setting 'xpack.monitoring.elasticsearch.username' from environment.
logstash  | Using bundled JDK: /usr/share/logstash/jdk
13

Thanks for update. this means will wait for next version. :slight_smile:

14

This is marked as fixed on 8.15.2, so I would comment on the Github PR that marked this is fixed.

It is this one: Fixes the issue where LS wipes out all quotes from docker env variables. by mashhurs · Pull Request #16456 · elastic/logstash · GitHub

15

@leandrojmp - Yes in the release notes page - Logstash 8.15.2 Release Notes | Logstash Reference [8.15] | Elastic

i could see that it is fixed, but i tried this morning it didn't work as expected like old version - 8.12.2.

16

@leandrojmp - Is it something that im missing with Logstash - 8.15.2?

17

Hi All, Since Logstash is functioning properly with v8.12.2 even though it is stated that it has been resolved with v8.15.2, But it is not working correctly as expected.

So, please let me know whether I should wait for the next release?

18

Did you report it again to Elastic by commenting on the closing issue or opening a new issue?

If you do not report it as a bug or unsolved, Elastic will not look into it.

You need to open a new issue or comment on this one with evidences that the fix didn't work.

19

Ok thanks @leandrojmp - Issue has been registered here [Logstash container fails to start · Issue #16563 · elastic/logstash · GitHub]