In attempting to debug why the Logstash 5.1.2 wasn't starting after an installation / upgrade using the Puppet module (https://github.com/elastic/puppet-logstash), I discovered some inconsistencies between the error messages displayed and the actual behavior of the service launcher.
Most of my issues are derived from attempting to clear this error message:
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
- from https://github.com/elastic/logstash/blob/master/logstash-core/lib/logstash/runner.rb#L169
To begin with, it makes mention of LS_HOME, which doesn't seem to align with any environment variable on Linux systems (though I see references to it in Windows batch scripts). Perhaps that's just a documentation issue, but let's continue anyway with LOGSTASH_HOME, which seems to be more correct from reading through the code.
It seems that LOGSTASH_HOME is assumed to be the path containing the application, in addition to patterns and plugins and the base path for the configuration file (according to the message above). However, this doesn't seem to allow for the /usr/share/logstash split from /etc/logstash (which previously held, and is apparently still expected to hold, the patterns, plugins, and configuration), as it's a single environment variable. Perhaps I'm misinterpreting the code paths as I trace through bin/logstash into lib/bootstrap/environment.rb and so on?
At no point can I find any sort of default that would make use of /etc/logstash as the above message seems to suggest, nor does it seem that I can utilize LOGSTASH_HOME cleanly. The message regarding --path.settings at least seems to offer some help, but that message doesn't seem to suggest that it's required in order to make use of configuration in /etc/logstash.
In addition, at least on Red Hat systems, it doesn't seem that any of the environment information in startup.options is actually passed through to the service. I've read the header regarding its usage in bin/system-install, but I would agree with the comments on https://github.com/elastic/logstash/issues/6482 that it's unexpected behavior for the environment information specified in startup.options to not get carried through to the service installation (either via /etc/sysconfig/logstash or /etc/defaults/logstash, which PleaseRun makes sure to reference in the systemd unit file). Thankfully, it seems work is being done on this front, thanks to the addition of environment options for PleaseRun!
As it stands, I've managed to dig through the code and find a workable solution, by setting LS_OPTS="--path.settings /etc/logstash" in startup.options and re-running system-install (and I suppose adding JAVA_HOME=/path/to/jdk directly to /etc/sysconfig/logstash for now), but I thought I should point out what I found to be some pretty annoying discrepancies between "expected" behavior (both based on previous releases, and the error messages displayed) and what was actually required.
Hopefully this helps someone else out there, and this process can be made a bit smoother in a future release.