I am using the logstash keystore for passwords in my logstash configs, .e.g
jdbc_password => "${DB_PWD}"
I have 20 pipelines where all configs use the keystore like this.
At startup I get random errors such as the ones below. Different pipelines crash each time. Sometimes no errors at all. The randomness makes be suspect that there is some issue with concurrent access to the keystore.
So, my first question is: is the access to the keystore thread safe? Can it handle concurrent access from multiple pipelines in the same logstash instance?
Elasticsearch version: 6.4.0 and 6.5.1.
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
The errors I am seeing:
[ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:mypipeline, :exception=>"Java::OrgLogstashSecretStore::SecretStoreException::UnknownException", :message=>"Error while trying to load the Logstash keystore", :backtrace=>["org.logstash.secret.store.backend.JavaKeyStore.load(org/logstash/secret/store/backend/JavaKeyStore.java:280)", "org.logstash.secret.store.backend.JavaKeyStore.load(org/logstash/secret/store/backend/JavaKeyStore.java:40)", "org.logstash.secret.store.SecretStoreFactory.doIt(org/logstash/secret/store/SecretStoreFactory.java:107)", "org.logstash.secret.store.SecretStoreFactory.load(org/logstash/secret/store/SecretStoreFactory.java:93)", "org.logstash.secret.store.SecretStoreExt.getIfExists(org/logstash/secret/store/SecretStoreExt.java:37)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:468)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:370)", "usr.share.logstash.logstash_minus_core.lib.logstash.util.substitution_variables.block in replace_placeholders(/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:45)", "org.jruby.RubyString.gsubCommon19(org/jruby/RubyString.java:2629)", "org.jruby.RubyString.gsubCommon19(org/jruby/RubyString.java:2583)", "org.jruby.RubyString.gsub(org/jruby/RubyString.java:2541)", "usr.share.logstash.logstash_minus_core.lib.logstash.util.substitution_variables.replace_placeholders(/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:35)", "usr.share.logstash.logstash_minus_core.lib.logstash.util.substitution_variables.deep_replace(/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:23)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.block in config_init(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:82)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1362)", "org.jruby.RubyHash$12.visit(org/jruby/RubyHash.java:1359)", "org.jruby.RubyHash.visitLimited(org/jruby/RubyHash.java:662)", "org.jruby.RubyHash.visitAll(org/jruby/RubyHash.java:647)", "org.jruby.RubyHash.iteratorVisitAll(org/jruby/RubyHash.java:1319)", "org.jruby.RubyHash.each_pairCommon(org/jruby/RubyHash.java:1354)", "org.jruby.RubyHash.each(org/jruby/RubyHash.java:1343)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.config_init(/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:81)", "usr.share.logstash.logstash_minus_core.lib.logstash.config.mixin.RUBY$method$config_init$0$__VARARGS__(usr/share/logstash/logstash_minus_core/lib/logstash/config//usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb)", "usr.share.logstash.logstash_minus_core.lib.logstash.outputs.base.initialize(/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:60)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1001)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.RubyClass.finvoke(org/jruby/RubyClass.java:908)", "org.jruby.RubyBasicObject.callMethod(org/jruby/RubyBasicObject.java:363)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.initialize(org/logstash/config/ir/compiler/OutputStrategyExt.java:224)", "org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt$INVOKER$i$1$0$initialize.call(org/logstash/config/ir/compiler/OutputStrategyExt$SimpleAbstractOutputStrategyExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1022)", "org.logstash.config.ir.compiler.OutputDelegatorExt.initialize(org/logstash/config/ir/compiler/OutputDelegatorExt.java:48)", "org.logstash.config.ir.compiler.OutputDelegatorExt.initialize(org/logstash/config/ir/compiler/OutputDelegatorExt.java:30)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:217)", "org.logstash.plugins.PluginFactoryExt$Plugins.plugin(org/logstash/plugins/PluginFactoryExt.java:166)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline.plugin(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:71)", "RUBY.<eval>((eval):87)", "org.jruby.RubyKernel.evalCommon(org/jruby/RubyKernel.java:1027)", "org.jruby.RubyKernel.eval(org/jruby/RubyKernel.java:994)", "org.jruby.RubyKernel$INVOKER$s$0$3$eval19.call(org/jruby/RubyKernel$INVOKER$s$0$3$eval19.gen)", "RUBY.initialize(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49)", "RUBY.initialize(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90)", "org.jruby.RubyClass.newInstance(org/jruby/RubyClass.java:1022)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(org/jruby/RubyClass$INVOKER$i$newInstance.gen)", "RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:38)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.block in converge_state(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:309)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:289)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:246)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2018-11-26T16:29:39,440][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::OrgLogstashSecretStore::SecretStoreException::UnknownException` for `PipelineAction::Create<egi_execution_batch>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:103:in `create'", "org/logstash/execution/ConvergeResultExt.java:34:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:321:in `block in converge_state'"]}