Logstash keystore encoding error

I try to set up a password protected Logstash keystore. It works when I run Logstash from the command line (as root, with set environment variable for the keystore), but when running as a systemd service (installed from deb package) it fails with:

Jul 31 14:00:06 my-server01 logstash[7380]: [2019-07-31T14:00:06,606][ERROR][logstash.config.sourceloader] Could not fetch all the sources {:exception=>LogStash::ConfigLoadingError, :message=>"The following config files contains non-ascii characters but are not UTF-8 encoded ["/etc/logstash/logstash.keystore"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/config/source/local.rb:83:in read'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/local.rb:94:inread'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/local.rb:190:in local_pipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source/local.rb:161:inpipeline_configs'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:61:in block in fetch'", "org/jruby/RubyArray.java:2572:incollect'", "/usr/share/logstash/logstash-core/lib/logstash/config/source_loader.rb:60:in fetch'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:148:inconverge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:inblock in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

Jul 31 14:00:06 my-server01 logstash[7380]: [2019-07-31T14:00:06,615][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: The following config files contains non-ascii characters but are not UTF-8 encoded ["/etc/logstash/logstash.keystore"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:155:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:96:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:367:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}

I created the keystore this way:

  • Created file /etc/sysconfig/logstash, owner root, permissions 600, with content:
    LOGSTASH_KEYSTORE_PASS=mypassword
  • On terminal as root:
    set +o history
    export LOGSTASH_KEYSTORE_PASS=mypassword
    set -o history
    sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash create
  • Add Elasticsearch password:
    sudo -E /usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash add ES_PWD

That is telling you that it is trying to read the keystore as part of the configuration. In case this is not clear. If I configure a pipeline using

- pipeline.id: main
  path.config: "/bin/bash"

I get the error message "The following config files contains non-ascii characters but are not UTF-8 encoded "/bin/bash"]"

So when you run as a service path.config is getting set to /etc/logstash. How are you setting path.config when you run from the command line?

This wasn't clear to me. So does this mean the location of the keystone file is wrong?

From the command line I set it as a parameter:

sudo -E /usr/share/logstash/bin/logstash --path.settings /etc/logstash -f /etc/logstash/conf.d/my-server01.conf

Another thing I noticed was that /etc/logstash had owner root after installation and it only would start after changing that to logstash:logstash.

No, it means the setting of path.config is wrong when your run as a service. path.config can be set using -f on the command line, or in logstash.yml, or, if you are using pipelines, in pipelines.yml.

If I set ExecStart in systemd to
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" "-f /etc/logstash"
it works like on the command line.

It's still a mystery to me why that is, however.