Logstash service issues (LOGSTASH_HOME / path.settings)

In attempting to debug why the Logstash 5.1.2 wasn't starting after an installation / upgrade using the Puppet module (https://github.com/elastic/puppet-logstash), I discovered some inconsistencies between the error messages displayed and the actual behavior of the service launcher.

Most of my issues are derived from attempting to clear this error message:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
- from https://github.com/elastic/logstash/blob/master/logstash-core/lib/logstash/runner.rb#L169

To begin with, it makes mention of LS_HOME, which doesn't seem to align with any environment variable on Linux systems (though I see references to it in Windows batch scripts). Perhaps that's just a documentation issue, but let's continue anyway with LOGSTASH_HOME, which seems to be more correct from reading through the code.

It seems that LOGSTASH_HOME is assumed to be the path containing the application, in addition to patterns and plugins and the base path for the configuration file (according to the message above). However, this doesn't seem to allow for the /usr/share/logstash split from /etc/logstash (which previously held, and is apparently still expected to hold, the patterns, plugins, and configuration), as it's a single environment variable. Perhaps I'm misinterpreting the code paths as I trace through bin/logstash into lib/bootstrap/environment.rb and so on?

At no point can I find any sort of default that would make use of /etc/logstash as the above message seems to suggest, nor does it seem that I can utilize LOGSTASH_HOME cleanly. The message regarding --path.settings at least seems to offer some help, but that message doesn't seem to suggest that it's required in order to make use of configuration in /etc/logstash.

In addition, at least on Red Hat systems, it doesn't seem that any of the environment information in startup.options is actually passed through to the service. I've read the header regarding its usage in bin/system-install, but I would agree with the comments on https://github.com/elastic/logstash/issues/6482 that it's unexpected behavior for the environment information specified in startup.options to not get carried through to the service installation (either via /etc/sysconfig/logstash or /etc/defaults/logstash, which PleaseRun makes sure to reference in the systemd unit file). Thankfully, it seems work is being done on this front, thanks to the addition of environment options for PleaseRun!

As it stands, I've managed to dig through the code and find a workable solution, by setting LS_OPTS="--path.settings /etc/logstash" in startup.options and re-running system-install (and I suppose adding JAVA_HOME=/path/to/jdk directly to /etc/sysconfig/logstash for now), but I thought I should point out what I found to be some pretty annoying discrepancies between "expected" behavior (both based on previous releases, and the error messages displayed) and what was actually required.

Hopefully this helps someone else out there, and this process can be made a bit smoother in a future release.

I wanted to go ahead and update my previous message with a few things that I've learned while continuing to poke around ...

Apparently the LS_SETTINGS_DIR environment variable allows setting /etc/logstash as the configuration root, which is used by the Logstash runner. Coincidentally, the Logstash Puppet module sets this variable in startup.options, though it doesn't yet get handed to the service (pending the updates to system-install).

Additionally, the maintainers of that Puppet module were kind enough to pre-set --path.settings in LS_OPTS ... which I would have noticed had I not been overriding LS_OPTS due to a legacy requirement (needing to increase the number of workers). So, many thanks to those individuals for trying to make this transparent for me!

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.