we recently started looking into deploying multiple instances of logstash on the same box and noticed that when the logstash settings directory is changed from the default (/etc/logstash), the jvm.options file is no longer read.
For example in our current setup we have:
/etc/logstash/instance1/(logstash.yml, jvm.options etc)
/etc/logstash/instance2/(logstash.yml, jvm.options etc)
and jvm.options is not detected.
If we move jvm.options back in /etc/logstash/ then it's read as normale (although neither instance has /etc/logstash/ has settings dir)
Looking into the bin/logstash.lib.sh I found a snippet of code where the /etc/logstash/jvm.options path is hardcoded:
if [ -z "$LS_JVM_OPTS" ]; then
for jvm_options in /etc/logstash/jvm.options
Is this a design decision (as the idea is it doesn't make sense to have a separate jvm.options for each installation running on the same box) or is it a bug?
Any help is appreciated
How did you install the instances? With the RPM/DEB package, or the TAR/ZIP package?
With the code you pasted, you can clearly see that Logstash expects to find the
jvm.options file in
/etc/logstash or in
$LOGSTASH_HOME/config. This is by design, as making
jvm.options deterministic proved exceptionally hard. How will
$LOGSTASH_HOME/bin/logstash.lib.sh know where to find
jvm.options if it is not hard coded or in the
$LOGSTASH_HOME path somewhere? How would you make finding it deterministic? I would be very open to adding in a deterministic way of doing this if you can find one, but this puzzle has eluded us. This is why you must set
$LS_JVM_OPTS on your own to avoid that block of code.
Provided you have a single install (from package or tarball) the current solution is to keep these settings in
$LOGSTASH_HOME/config, as they are searched in that order. If the values in
jvm.options need to be different for different instances, you should probably consider having multiple tarball instances of Logstash on the same machine, so that each will have its own
$LOGSTASH_HOME, and therefore can have separate configurations. You could symlink
/etc/logstash/instanceN to the respective paths. You can also use
$LOGSTASH_HOME/bin/system-install $LOGSTASH_HOME/config/startup.options to create separate SysV/upstart/systemd startup scripts for each instance (be sure to edit
Thanks for your reply.
the installation has been done using Ansible to deploy the rpm packaged logstash and deleting the standard scripts/setting files
After that templated copies of the logstash.yml, systemd scripts and default files are deployed for each instance.
All the instances have the same LOGSTASH_HOME, but each of the has a separate LS_SETTINGS_DIR with logstash.yml etc in it, a separate systemd service definition and a separate /etc/sysconfig/instance_X_logstash file where environment variables are defined.
I agree I wouldn't know how to make it deterministic.
I was wondering though why logstash doesn't try to load jvm.options also from the path specified in LS_SETTINGS_DIR, as that is what I was expecting after reading the directory layout documentation https://www.elastic.co/guide/en/logstash/current/dir-layout.html, where it says the settings directory contains: "Configuration files, including logstash.yml and jvm.options"
Thank you in advance
That's a bitter irony, isn't it?
The problem is that because Logstash needs Java to run, it needs its Java settings to be set before it can load any of the other configuration data. That means that it doesn't know where
$LS_SETTINGS_DIR is until after it's actually started the startup process and reads the other configuration files. It can easily determine where
$LS_HOME is, but what if
$LS_SETTINGS_DIR is not a subdirectory of
If you are setting environment variables in
/etc/sysconfig/instance_X_logstash, you might try to set
$LS_JVM_OPTS in there, having it point to the location of the respective
jvm.options file. It's supposed to use that, as stated in my previous post.
I should point out here that this (not reading
jvm.options from the
$LS_SETTINGS_DIR) is resolved and will ship in Logstash 5.2.0: https://github.com/elastic/logstash/pull/6382
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.