Not a valid Logstash keystore java exception problem

Elastic Stack Logstash
1

hello
i encrypted the username and password of my elasticsearch and call them as variables in the pipeline.conf but I wasn't able to get it to work properly

here is the guide I followed:
Secrets keystore for secure settings | Logstash Reference [8.15] | Elastic

here is the command I used
to create my keystore:

set +o history
export LOGSTASH_KEYSTORE_PASS=mypassword
set -o history
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create

and then to add my variables:

sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_USER
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_PWD

and then list them:

sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list

I gave the logstash.keystore file the appropriate permissions/user.group
and in the pipeline.conf and yml files use

output {
  elasticsearch {
    ...
    user => "${ES_USER}"
    password => "${ES_PWD}"
    ...
  }
}

here is the the output of the keystone creation :

root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:29:55,998 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:29:55,999 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:29:55,999 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:29:56,000 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

[2024-08-22T01:29:56,574][INFO ][org.logstash.secret.store.backend.JavaKeyStore] Created Logstash keystore at /etc/logstash/logstash.keystore
Created Logstash keystore at /etc/logstash/logstash.keystore
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_USER
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:30:14,473 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:30:14,474 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:30:14,475 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:30:14,475 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

Enter value for ES_USER:
Added 'es_user' to the Logstash keystore.
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_PWD
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:31:01,325 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:31:01,326 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:31:01,327 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:31:01,327 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

Enter value for ES_PWD:
Added 'es_pwd' to the Logstash keystore.
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:31:51,318 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:31:51,319 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:31:51,320 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:31:51,320 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties

es_pwd
es_user
root@logstash:/etc/logstash# ls
'${sys:ls.logs}'   certs   conf.d   jvm.options   log4j2.properties   logstash.keystore   logstash-sample.conf   logstash.yml   pipelines.yml   startup.options
root@logstash:/etc/logstash# chown root:logstash logstash.keystore
root@logstash:/etc/logstash# chmod 770 logstash.keystore

finally here is the error I got in logstash log journal :

root@logstash:/etc/logstash/conf.d# systemctl restart logstash
root@logstash:/etc/logstash/conf.d# journalctl -u logstash -f
-- Logs begin at Sun 2024-08-18 20:34:53 UTC. --
Aug 22 01:38:00 logstash logstash[886]: }
Aug 22 01:38:05 logstash systemd[1]: Stopping logstash...
Aug 22 01:38:05 logstash logstash[886]: [2024-08-22T01:38:05,217][WARN ][logstash.runner          ] SIGTERM received. Shutting down.
Aug 22 01:38:06 logstash logstash[886]: [2024-08-22T01:38:06,606][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
Aug 22 01:38:07 logstash logstash[886]: [2024-08-22T01:38:07,318][INFO ][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
Aug 22 01:38:07 logstash logstash[886]: [2024-08-22T01:38:07,352][INFO ][logstash.runner          ] Logstash shut down.
Aug 22 01:38:07 logstash systemd[1]: logstash.service: Succeeded.
Aug 22 01:38:07 logstash systemd[1]: Stopped logstash.
Aug 22 01:38:07 logstash systemd[1]: Started logstash.
Aug 22 01:38:07 logstash logstash[41720]: Using bundled JDK: /usr/share/logstash/jdk
Aug 22 01:38:25 logstash logstash[41720]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,120][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,123][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.15.0", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,126][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms4g, -Xmx4g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,130][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,130][INFO ][logstash.runner          ] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,040][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::OrgLogstashSecretStore::SecretStoreException::LoadException", :message=>"Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore.", :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.load(JavaKeyStore.java:294)", "org.logstash.secret.store.backend.JavaKeyStore.load(JavaKeyStore.java:77)", "org.logstash.secret.store.SecretStoreFactory.doIt(SecretStoreFactory.java:129)", "org.logstash.secret.store.SecretStoreFactory.load(SecretStoreFactory.java:115)", "org.logstash.secret.store.SecretStoreExt.getIfExists(SecretStoreExt.java:60)", "org.logstash.execution.AbstractPipelineExt.getSecretStore(AbstractPipelineExt.java:797)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:238)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:173)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1379)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:949)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"]}
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,076][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,085][INFO ][logstash.runner          ] Logstash shut down.
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,092][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
Aug 22 01:38:27 logstash logstash[41720]: org.jruby.exceptions.SystemExit: (SystemExit) exit
Aug 22 01:38:27 logstash logstash[41720]:         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
Aug 22 01:38:27 logstash logstash[41720]:         at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
Aug 22 01:38:27 logstash logstash[41720]:         at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Failed with result 'exit-code'.
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
Aug 22 01:38:27 logstash systemd[1]: Stopped logstash.
Aug 22 01:38:27 logstash systemd[1]: Started logstash.

please suggest

2

The logstash instance running as a service needs access to this password. That may require adding it to /etc/sysconfig/logstash

See here and here.

1 Like
3

Thank you somuch badger

It worked !

here is what i did

before creating the keystore and adding the secret username and password
i went and creat the directory /etc/sysconfig/ and a logstash file in it with the value of LOGSTASH_KEYSTORE_PASS
here are the commands :

sudo systemctl stop logstash.service 
sudo mkdir /etc/sysconfig
sudo nano /etc/sysconfig/logstash
add the following (replace with actual password): 
LOGSTASH_KEYSTORE_PASS=mypassword
sudo chmod 600 /etc/sysconfig/logstash

Then created my keystore and continued with the steps as described above

but i'm still wondring, it is secure to have a file storing such a value in sysconfig ?

4

In many cases it's the best you can do. At the high-end (e.g. in financial services) you can spend tens of thousands of dollars on an HSM (hardware security module) connected to your server and store the password there. At the low-end you can spend fifty bucks on a USB device to connect to your laptop.

But for a cloud-hosted server you either wait for someone to log in and type the password after the server boots, or you store the plaintext password on the disk (if you encrypt it you need the decryption key stored on the disk too). Basically you have to trust folks who have privileged access to your servers, and no amount of obfuscation changes that.

1 Like