Setting jvm.options with multiple logstash (5.x) instances on the same box

Elastic Stack Logstash
1

Hi all,

we recently started looking into deploying multiple instances of logstash on the same box and noticed that when the logstash settings directory is changed from the default (/etc/logstash), the jvm.options file is no longer read.

For example in our current setup we have:
/etc/logstash/instance1/(logstash.yml, jvm.options etc)
/etc/logstash/instance2/(logstash.yml, jvm.options etc)

and jvm.options is not detected.
If we move jvm.options back in /etc/logstash/ then it's read as normale (although neither instance has /etc/logstash/ has settings dir)

Looking into the bin/logstash.lib.sh I found a snippet of code where the /etc/logstash/jvm.options path is hardcoded:

if [ -z "$LS_JVM_OPTS" ]; then
for jvm_options in /etc/logstash/jvm.options
"$LOGSTASH_HOME"/config/jvm.options;
do
...
Is this a design decision (as the idea is it doesn't make sense to have a separate jvm.options for each installation running on the same box) or is it a bug?

Any help is appreciated
Thanks

2

How did you install the instances? With the RPM/DEB package, or the TAR/ZIP package?

With the code you pasted, you can clearly see that Logstash expects to find the jvm.options file in /etc/logstash or in $LOGSTASH_HOME/config. This is by design, as making jvm.options deterministic proved exceptionally hard. How will $LOGSTASH_HOME/bin/logstash.lib.sh know where to find jvm.options if it is not hard coded or in the $LOGSTASH_HOME path somewhere? How would you make finding it deterministic? I would be very open to adding in a deterministic way of doing this if you can find one, but this puzzle has eluded us. This is why you must set $LS_JVM_OPTS on your own to avoid that block of code.

Provided you have a single install (from package or tarball) the current solution is to keep these settings in /etc/logstash or $LOGSTASH_HOME/config, as they are searched in that order. If the values in jvm.options need to be different for different instances, you should probably consider having multiple tarball instances of Logstash on the same machine, so that each will have its own $LOGSTASH_HOME, and therefore can have separate configurations. You could symlink /etc/logstash/instanceN to the respective paths. You can also use $LOGSTASH_HOME/bin/system-install $LOGSTASH_HOME/config/startup.options to create separate SysV/upstart/systemd startup scripts for each instance (be sure to edit startup.options accordingly).

3

Hi Aaron,
Thanks for your reply.

the installation has been done using Ansible to deploy the rpm packaged logstash and deleting the standard scripts/setting files

After that templated copies of the logstash.yml, systemd scripts and default files are deployed for each instance.
All the instances have the same LOGSTASH_HOME, but each of the has a separate LS_SETTINGS_DIR with logstash.yml etc in it, a separate systemd service definition and a separate /etc/sysconfig/instance_X_logstash file where environment variables are defined.

I agree I wouldn't know how to make it deterministic.
I was wondering though why logstash doesn't try to load jvm.options also from the path specified in LS_SETTINGS_DIR, as that is what I was expecting after reading the directory layout documentation https://www.elastic.co/guide/en/logstash/current/dir-layout.html, where it says the settings directory contains: "Configuration files, including logstash.yml and jvm.options"

Thank you in advance

4

That's a bitter irony, isn't it?

The problem is that because Logstash needs Java to run, it needs its Java settings to be set before it can load any of the other configuration data. That means that it doesn't know where $LS_SETTINGS_DIR is until after it's actually started the startup process and reads the other configuration files. It can easily determine where $LS_HOME is, but what if $LS_SETTINGS_DIR is not a subdirectory of $LS_HOME?

If you are setting environment variables in /etc/sysconfig/instance_X_logstash, you might try to set $LS_JVM_OPTS in there, having it point to the location of the respective jvm.options file. It's supposed to use that, as stated in my previous post.

1 Like
5

I should point out here that this (not reading jvm.options from the $LS_SETTINGS_DIR) is resolved and will ship in Logstash 5.2.0: https://github.com/elastic/logstash/pull/6382

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.