Logstash not reading config file when ran as a service

Elastic Stack Logstash
1

Hello again all. Thank you so much for your help on my previous question.
Hopefully this is my last (lol almost got that out with a straight face)

When I run the command line on my redhat box

./logstash -f /usr/share/logstash/unique_name.conf
Everything works as planned.

I have copied this file into the /etc/logstash/conf.d directory.
However.. when I run initctl start logstash the process runs but it does not appear that my config file is being processed. I can do a ps-ef|grep logstash and see the output.
did I miss editing a file or do I need to put this conf file somewhere else? Please advise and thank you !

(proof the service is running)

ps -ef|grep logstash

logstash 6117 1 53 14:02 ? 00:00:32 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash
root 6152 5037 0 14:03 pts/2 00:00:00 grep logstash

my config file in /etc/logstash/conf.d directory.

more unique_name.conf

input {
file {
path => "/logs/perf/syslog_perf.log"
start_position => "beginning"
}
}

output {
elasticsearch {
hosts => ["localhost:9200"]
}
stdout { codec => rubydebug }
}

2

Is data being added to /logs/perf/syslog_perf.log? Does the logstash user have read access to that file?

3

Yes I see the file being written to .
my permissions are set up for
-rw-r--r--. 1 root root 215439877 Apr 26 07:23 syslog_perf.log

4

What about the permissions of /logs and /logs/perf?

You can increase Logstash's log level to get more clues about what it's doing. Permission problems should be quite visible.

5

drwxr-xr-x. 5 root root 4096 Apr 25 07:41 logs

drwxr-xr-x. 2 root root 4096 Apr 25 09:44 perf

I do see a difference between when I run the command as root fro mthe command line and when I run it with intictl

here is root from the command line
ps -ef | grep logstash
root 6380 1 4 Apr25 ? 00:42:27 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb -f /usr/share/logstash/rsyslog_bwc.conf

here is at initctl start logstash
ps -ef | grep logstash
logstash 13433 1 99 07:39 ? 00:00:09 /usr/bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -Djava.awt.headless=true -Dfile.encoding=UTF-8 -XX:+HeapDumpOnOutOfMemoryError -Xmx1g -Xms256m -Xss2048k -Djffi.boot.library.path=/usr/share/logstash/vendor/jruby/lib/jni -Xbootclasspath/a:/usr/share/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/usr/share/logstash/vendor/jruby -Djruby.lib=/usr/share/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash

Perhaps the path.settings is not picking up my conf file?
I'm uncertain.

6

Perhaps the path.settings is not picking up my conf file?

Yes, unless you're setting path.config to /usr/share/logstash Logstash won't look for configuration files there.

7

How can i check and modify the path.config? Thank you for your help magnus ! :slight_smile:

8

Look in logstash.yml in the directory pointed to by --path.settings.

9

I show the following in the logstash.yml

------------ Pipeline Configuration Settings --------------

Where to fetch the pipeline configuration for the main pipeline

path.config: /etc/logstash/conf.d

I have the following files (both are the same config located in the /etc/logstash.conf.d folder
-rw-r--r--. 1 root root 197 Apr 25 13:57 default.conf
-rw-r--r--. 1 root root 197 Apr 25 13:38 rsyslog_bwc.conf

10

Okay, this looks fine. As I said, increase the logging verbosity to find out more about what's happening.

11

how can i do that when issuing the initctl start logstash command?
sorry for such a basic question... do i pass a --v at the end ? im unfamiliar with the syntax

12

The log level is configurable in logstash.yml, see https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html.

13

I have set the level to debug.

------------ Debugging Settings --------------

Options for log.level:

* fatal

* error

* warn

* info (default)

* debug

* trace

log.level: info

log.level: debug
path.logs: /var/log/logstash

The only messages I see in the /var/log/logstash/logstash-plain.log have occurred when I run with initctl.

[2017-04-26T08:53:53,961][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:420:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:202:invalidate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:118:in validate_all'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:117:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:210:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:183:inrun'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}
[

14

Is that the last thing in the log? Because it indicates that Logstash doesn't start at all.

15

yes... could there be a permission issue? with writing?

is there an easy way to test the logstash user account access compared to running it with sudo at the command line?

16

It looks like a config file validation problem, but with the way you posted the log snippet some of the interesting parts were hidden. What comes after {:error=>#?

17

[2017-04-26T09:42:07,897][FATAL][logstash.runner ]
An unexpected error occurred! {:error=>#<ArgumentError: Path "/usr/share/logstash/data/queue" must be a writable directory.
It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:420:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:202:invalidate_value'",
"/usr/share/logstash/logstash-core/lib/logstash/settings.rb:118:in validate_all'", "org/jruby/RubyHash.java:1342:ineach'",
"/usr/share/logstash/logstash-core/lib/logstash/settings.rb:117:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:210:inexecute'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:183:inrun'",
"/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in(root)'"]}

18

And what are your thoughts about that error message?

19

yup.... needs writability let

20

Thank you Kindly Magnus for your help on this matter :slight_smile: