# Not a valid Logstash keystore java exception problem
**URL:** https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362
**Category:** Logstash
**Created:** [August 22, 2024, 9:51am UTC](https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362 "2024-08-22T09:51:54Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Abdarrahmane](https://discuss.elastic.co/u/Abdarrahmane)
#### Post date: [August 22, 2024, 9:51am UTC](https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362/1 "2024-08-22T09:51:54Z")
hello
i encrypted the username and password of my elasticsearch and call them as variables in the `pipeline.conf` but I wasn't able to get it to work properly
here is the guide I followed:
[Secrets keystore for secure settings | Logstash Reference [8.15] | Elastic](https://www.elastic.co/guide/en/logstash/current/keystore.html)
here is the command I used
to create my keystore:
```auto
set +o history
export LOGSTASH_KEYSTORE_PASS=mypassword
set -o history
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
```
and then to add my variables:
```auto
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_USER
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_PWD
```
and then list them:
```auto
sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
```
I gave the `logstash.keystore` file the appropriate permissions/user.group
and in the pipeline.conf and yml files use
```auto
output {
elasticsearch {
...
user => "${ES_USER}"
password => "${ES_PWD}"
...
}
}
```
here is the the output of the keystone creation :
```auto
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:29:55,998 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:29:55,999 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:29:55,999 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:29:56,000 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
[2024-08-22T01:29:56,574][INFO][org.logstash.secret.store.backend.JavaKeyStore] Created Logstash keystore at /etc/logstash/logstash.keystore
Created Logstash keystore at /etc/logstash/logstash.keystore
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_USER
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:30:14,473 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:30:14,474 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:30:14,475 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:30:14,475 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Enter value for ES_USER:
Added 'es_user' to the Logstash keystore.
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_PWD
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:31:01,325 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:31:01,326 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:31:01,327 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:31:01,327 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Enter value for ES_PWD:
Added 'es_pwd' to the Logstash keystore.
root@logstash:/etc/logstash# sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash list
Using bundled JDK: /usr/share/logstash/jdk
2024-08-22 01:31:51,318 main ERROR Unable to locate appender "${sys:ls.log.format}_console" for logger config "root"
2024-08-22 01:31:51,319 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling" for logger config "root"
2024-08-22 01:31:51,320 main ERROR Unable to locate appender "${sys:ls.log.format}_rolling_slowlog" for logger config "slowlog"
2024-08-22 01:31:51,320 main ERROR Unable to locate appender "${sys:ls.log.format}_console_slowlog" for logger config "slowlog"
Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
es_pwd
es_user
root@logstash:/etc/logstash# ls
'${sys:ls.logs}' certs conf.d jvm.options log4j2.properties logstash.keystore logstash-sample.conf logstash.yml pipelines.yml startup.options
root@logstash:/etc/logstash# chown root:logstash logstash.keystore
root@logstash:/etc/logstash# chmod 770 logstash.keystore
```
finally here is the error I got in logstash log journal :
```auto
root@logstash:/etc/logstash/conf.d# systemctl restart logstash
root@logstash:/etc/logstash/conf.d# journalctl -u logstash -f
-- Logs begin at Sun 2024-08-18 20:34:53 UTC. --
Aug 22 01:38:00 logstash logstash[886]: }
Aug 22 01:38:05 logstash systemd[1]: Stopping logstash...
Aug 22 01:38:05 logstash logstash[886]: [2024-08-22T01:38:05,217][WARN][logstash.runner] SIGTERM received. Shutting down.
Aug 22 01:38:06 logstash logstash[886]: [2024-08-22T01:38:06,606][INFO][logstash.javapipeline][main] Pipeline terminated {"pipeline.id"=>"main"}
Aug 22 01:38:07 logstash logstash[886]: [2024-08-22T01:38:07,318][INFO][logstash.pipelinesregistry] Removed pipeline from registry successfully {:pipeline_id=>:main}
Aug 22 01:38:07 logstash logstash[886]: [2024-08-22T01:38:07,352][INFO][logstash.runner] Logstash shut down.
Aug 22 01:38:07 logstash systemd[1]: logstash.service: Succeeded.
Aug 22 01:38:07 logstash systemd[1]: Stopped logstash.
Aug 22 01:38:07 logstash systemd[1]: Started logstash.
Aug 22 01:38:07 logstash logstash[41720]: Using bundled JDK: /usr/share/logstash/jdk
Aug 22 01:38:25 logstash logstash[41720]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,120][INFO][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,123][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"8.15.0", "jruby.version"=>"jruby 9.4.8.0 (3.1.4) 2024-07-02 4d41e55a67 OpenJDK 64-Bit Server VM 21.0.4+7-LTS on 21.0.4+7-LTS +indy +jit [x86_64-linux]"}
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,126][INFO][logstash.runner] JVM bootstrap flags: [-Xms4g, -Xmx4g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dlogstash.jackson.stream-read-constraints.max-string-length=200000000, -Dlogstash.jackson.stream-read-constraints.max-number-length=10000, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED, -Dio.netty.allocator.maxOrder=11]
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,130][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-string-length` configured to `200000000`
Aug 22 01:38:26 logstash logstash[41720]: [2024-08-22T01:38:26,130][INFO][logstash.runner] Jackson default value override `logstash.jackson.stream-read-constraints.max-number-length` configured to `10000`
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,040][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::OrgLogstashSecretStore::SecretStoreException::LoadException", :message=>"Found a file at /etc/logstash/logstash.keystore, but it is not a valid Logstash keystore.", :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.load(JavaKeyStore.java:294)", "org.logstash.secret.store.backend.JavaKeyStore.load(JavaKeyStore.java:77)", "org.logstash.secret.store.SecretStoreFactory.doIt(SecretStoreFactory.java:129)", "org.logstash.secret.store.SecretStoreFactory.load(SecretStoreFactory.java:115)", "org.logstash.secret.store.SecretStoreExt.getIfExists(SecretStoreExt.java:60)", "org.logstash.execution.AbstractPipelineExt.getSecretStore(AbstractPipelineExt.java:797)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:238)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:173)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1379)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:949)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:363)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:118)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:354)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:1583)"]}
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,076][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,085][INFO][logstash.runner] Logstash shut down.
Aug 22 01:38:27 logstash logstash[41720]: [2024-08-22T01:38:27,092][FATAL][org.logstash.Logstash] Logstash stopped processing because of an error: (SystemExit) exit
Aug 22 01:38:27 logstash logstash[41720]: org.jruby.exceptions.SystemExit: (SystemExit) exit
Aug 22 01:38:27 logstash logstash[41720]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:921) ~[jruby.jar:?]
Aug 22 01:38:27 logstash logstash[41720]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:880) ~[jruby.jar:?]
Aug 22 01:38:27 logstash logstash[41720]: at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Failed with result 'exit-code'.
Aug 22 01:38:27 logstash systemd[1]: logstash.service: Scheduled restart job, restart counter is at 1.
Aug 22 01:38:27 logstash systemd[1]: Stopped logstash.
Aug 22 01:38:27 logstash systemd[1]: Started logstash.
```
please suggest
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 22, 2024, 12:53pm UTC](https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362/2 "2024-08-22T12:53:35Z")
> [@Abdarrahmane](#):
>
> `export LOGSTASH_KEYSTORE_PASS=mypassword`
The logstash instance running as a service needs access to this password. That may require adding it to /etc/sysconfig/logstash
See [here](https://www.elastic.co/guide/en/logstash/current/keystore.html#keystore-password) and [here](https://github.com/elastic/logstash/issues/9711).
---
### Author:  [@Abdarrahmane](https://discuss.elastic.co/u/Abdarrahmane)
#### Post date: [August 22, 2024, 3:55pm UTC](https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362/3 "2024-08-22T15:55:40Z")
Thank you somuch badger
It worked !
here is what i did
before creating the keystore and adding the secret username and password
i went and creat the directory `/etc/sysconfig/` and a logstash file in it with the value of `LOGSTASH_KEYSTORE_PASS`
here are the commands :
```auto
sudo systemctl stop logstash.service
sudo mkdir /etc/sysconfig
sudo nano /etc/sysconfig/logstash
add the following (replace with actual password):
LOGSTASH_KEYSTORE_PASS=mypassword
sudo chmod 600 /etc/sysconfig/logstash
```
Then created my keystore and continued with the steps as described above
but i'm still wondring, it is secure to have a file storing such a value in sysconfig ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 22, 2024, 4:34pm UTC](https://discuss.elastic.co/t/not-a-valid-logstash-keystore-java-exception-problem/365362/4 "2024-08-22T16:34:53Z")
> [@Abdarrahmane](#):
>
> it is secure to have a file storing such a value in sysconfig ?
In many cases it's the best you can do. At the high-end (e.g. in financial services) you can spend tens of thousands of dollars on an HSM (hardware security module) connected to your server and store the password there. At the low-end you can spend fifty bucks on a USB device to connect to your laptop.
But for a cloud-hosted server you either wait for someone to log in and type the password after the server boots, or you store the plaintext password on the disk (if you encrypt it you need the decryption key stored on the disk too). Basically you have to trust folks who have privileged access to your servers, and no amount of obfuscation changes that.