RPM packaging problems for CentOS 6/7

Hi,

I've noticed that new packaging system (tested rc1) has probably few flaws:

On CentOS6:

  • "service logstash stop/start" doesn't work anymore due to being handled by upstart start/stop, which breaks most automation in a great way (I ended up deploying my own "dummy" /etc/init.d/logstash to workaround this),
  • looks like settings from /etc/logstash/startup.options are completely ignored (yes, I've ran system-install)

On CentOS7:

  • systemd files are created with: EnvironmentFile=-/etc/default/logstash EnvironmentFile=-/etc/sysconfig/logstash but logstash isn't shipped with them anymore and they aren't generated by anything either
  • again, settings from /etc/logstash/startup.options are completely ignored (yes, I've ran system-install)

Is all this anyhow expected/fixable?

Also it's sligthly inconvienient that "system-install" script deletes its logs immediately. Couldn't this be saved somewhere under /var/lib/logstash on overwritten on every run?

Logstash detects that upstart is in use and makes upstart scripts. sudo initctl start|stop|restart logstash works quite well for automation.

Those entries in the systemd files are artifacts produced by pleaserun. Logstash uses its own configuration defaults in /etc/logstash/logstash.yml now, rather than /etc/default, so there's no reason to read from /etc/default/logstash.

What makes you say this? This has been tested fairly extensively, so I'd be interested to hear about your results. Are you running it as root/sudo?

It was only ever designed to show output in the result of a failure, rather than preserve results. The temp-file is randomly created, so to prevent random files lying around, it cleans up afterward. Feel free to comment out or change this for troubleshooting.

It doesn't look like either NICE or OPEN_FILES limits are applied:

$ sudo su -

# egrep "^LS_NICE|^LS_OPEN" /etc/logstash/startup.options 
LS_OPEN_FILES=262143
LS_NICE=19

# /usr/share/logstash/bin/system-install 
Sending all logs to /tmp/tmp.d1i27IuYZk
Successfully created system startup script for Logstash

# service logstash restart
Redirecting to /bin/systemctl restart  logstash.service

# ps -o ni -p $(pgrep -u logstash java)
 NI
  0

# cat /proc/$(pgrep -u logstash java)/limits | grep files
Max open files            4096                 4096                 files     

Both /etc/systemd/system/logstash.service (CentOS7) and /etc/init/logstash.conf (CentOS6) have no signs of settings those variables.

Also one more question: it looks like /etc/systemd/system/logstash.service is created directly in that dir. It looks like common practices for virtually all other services is to link from /etc/systemd/system/$foo.service to /usr/lib/systemd/system/$foo.services. Shouldn't logstash be no different here?

As you can see here, the values are passed to pleaserun. But you have also noticed that the values do not appear in /etc/systemd/system/logstash.service. This is because systemd unit files do not assign nice values or allow establishment of file handle limits. File handles and nice values (per user) should be handled in /etc/security/limits.conf for the Logstash user:

logstash   -       nofile          262143
logstash   -       nice            19

The same approach should be used in all startup forms, actually: init, upstart, and systemd. For upstart, uncommenting the appropriate lines in /etc/init/logstash may yield the desired result, but are still not preferred over settings in /etc/security/limits.conf. Editing of this file by Logstash at install time is not going to be well received by security professionals, so we leave it to the end user.

Putting custom services directly in this directory is the only way to guarantee that the service is recognized across platforms (RPM platforms, DEB platforms, etc). This is actually done by pleaserun, because it's the only commonly searched path across all known distributions using systemd. It took us a considerable amount of digging to find and verify this.

This syntax in systemd units means that it will attempt to load these files as environment files only if they exist. If they don't exist, nothing happens. This code exists for backwards compatibility and does nothing if you have neither of those files present.

Hmm. This is a limitation in pleaserun, right now. The template pleaserun uses to generate systemd units doesn't have support for Nice= or LimitNOFile=.

I have filed https://github.com/jordansissel/pleaserun/issues/120 track resolving this in pleaserun.

In the meantime, you can work around this by setting any resource values (nice, file limits, etc) in the systemd service yourself.

I have huge doubts about that. Quoting your own website: https://www.elastic.co/guide/en/elasticsearch/reference/master/setting-system-settings.html

However, systems which use systemd require that system limits are specified in a systemd configuration file.

Also quoting Kay Sievers from 2011: "Systemd does not support global limits, the file is intentionally ignored."

And finally my own tests:

CentOS 7:

# grep ^logstash /etc/security/limits.conf 
logstash   -       nofile          262143
logstash   -       nice            19
# service logstash restart
Redirecting to /bin/systemctl restart  logstash.service
# ps -o ni -p $(pgrep -u logstash java)
 NI
  0
# cat /proc/$(pgrep -u logstash java)/limits | grep files
Max open files            4096                 4096                 files  

It also doesn't seem to have an effect with CentOS6/upstart combo.

Not to mention that limits per user can be somewhat unpractical when you want to run logstash from root or shared user, which might be required if you use file input, especially with remote filesystems like NFS.

I stand corrected. My digging never did uncover those settings for systemd. We'll need to get this into a patch relatively quickly, I think.

hi @jordansissel

It looks like this needs to be added for upstart as well http://bryanmarty.com/2012/02/10/setting-nofile-limit-upstart/