Logstash service started but not working, command line is working perfectly

Elastic Stack Logstash
1

Dear,

I have a logstash configuration that is working when I run logstash manually :

sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d

But doesn't work when I'm running at as a service

service logstash start

To be clear, the service is able to start, no errors are printed in any logstash log file.
What concerns me a bit is that I do not have the file : /var/logs/logstash/logstash.log I only have

  • /var/log/logstash/logstash.err
  • /var/log/logstash/logstash.stdout

How the process looks when started manually and therefor is working :

ps -ef |grep logstash
root 25366 8161 0 10:23 pts/1 00:00:00 sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d
logstash 25367 25366 56 10:23 pts/1 00:00:16 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d

How the process looks when started via a service and therefor is not working :

logstash 25678 1 0 10:26 pts/1 00:00:00 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -Xmx500m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Djava.io.tmpdir=/var/lib/logstash -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log

Logstash has been installed using the rpm (version 2.1.1).

All files under /etc/logstash are owned by root , but are readable by the user logstash. All files and folders under /opt/logstash and /var/log/logstash are owned by the user logstash.

Result of the config test :

[root@jed logstash]# /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d --configtest
Configuration OK

Regards
Karel.

2

There is a workaround for this issue and that is to modify the init.d logstash script.
Put the lines that execute the program in command and add the following line before the lookup of the PID

sudo -u logstash /opt/logstash/bin/logstash agent -f /etc/logstash/conf.d -l ${LS_LOG_FILE} &