Keystore not working properly

[bkasrai]

Good afternoon,
I have been combing the threads here and following the documentation on setting up keystore however nothing in the threads seem to have a resolution and for whatever reason, even with the steps I am following per the documentation, I am still unable to get keystore working. I am looking to pull out the plain text password within the Elasticsearch output plugin when sending data over to ES, hence the need for keystore. Please see the steps below along with my output for each step. (this is also the link I followed-https://www.elastic.co/guide/en/logstash/current/keystore.html#keystore-password)

set o+ history export LOGSTASH_KEYSTORE_PASS=elastic_test_p@ssw0rd
set o- history
./logstash-keystore create es_password --path.settings /etc/logstash/

An Logstash keystore already exists. Overwrite ? [y/N] y
WARNING: The keystore password is not set. Please set the environment variable `LOGSTASH_KEYSTORE_PASS`. Failure to do so will result in reduced security. Continue without password protection on the keystore?
y
./logstash-keystore list --path.settings /etc/logstash/

Nothing shows up but I do see the logstash.keystore file where specified. I also have the Environment variable file created sitting in a file called logstash within the /etc/sysconfig directory. (proper permissions and all) When I go to test this within a conf.d file, it indicates it cannot find my keystore file and I am placing the variable as such in the output plugin

 user => "elastic"
       password => "${es_password}"

Can someone please walk me through where I am going wrong with this? The version of logstash I am using is 6.6. Thanks.

[bkasrai]

Here is the exact error when testing-

agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Cannot evaluate${es_password}. Replacement variablees_passwordis not defined in a Logstash secret store or as an Environment entry and there is no default value given.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:50:inblock in replace_placeholders'", "org/jruby/RubyString.java:2541:in gsub'", "/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:35:inreplace_placeholders'", "/usr/share/logstash/logstash-core/lib/logstash/util/substitution_variables.rb:23:in deep_replace'", "/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:82:inblock in config_init'", "org/jruby/RubyHash.java:1343:in each'", "/usr/share/logstash/logstash-core/lib/logstash/config/mixin.rb:81:inconfig_init'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:60:in initialize'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:224:ininitialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:48:in initialize'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:30:ininitialize'", "org/logstash/plugins/PluginFactoryExt.java:235:in plugin'", "org/logstash/plugins/PluginFactoryExt.java:181:inplugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:71:in plugin'", "(eval):13:in'", "org/jruby/RubyKernel.java:994:in eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:ininitialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:inblock in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:148:insynchronize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in exclusive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:327:in block in converge_state'"]}

[PhaedrusTheGreek (Jay Greenberg)]

I was able to reproduce this error under the following conditions:

1. The keystore file didn't have the correct permissions (owned by the logstash process user)
2. The keystore file didn't exist.

You can evaluate further by running logstash in debug mode and examining the logs, which should tell you the path of the keystore.