Logstash service/init is not doing anything

Elastic Stack Logstash
1

Hello All,

I have installed logstash using RPM. It created sample file in /etc/init/logstash.conf. When I start the process using sudo initctl start logstash, it is creating pid and the service is running in the background. But it does not seem to do anything. Overall, I have two issues now

  1. When the process is running it is not doing anything
  2. Right now (after fine tuning some settings?), it looks like it respawning the process.

Sample log for 2nd issue, from /var/log/messages

Feb 13 11:24:45 myhost init: logstash main process (5944) terminated with status 1
Feb 13 11:24:45 myhost init: logstash main process ended, respawning
Feb 13 11:25:05 myhost  init: logstash main process (6010) terminated with status 1
Feb 13 11:25:05 myhost init: logstash main process ended, respawning

As far as issue-1 is concerned, I am not seeing any updates under the logstash logs folder (/var/log/logstash/)

When i run the command using command line then it is ingesting the data (doing some action at least).
/usr/share/logstash/bin/logstash -f /usr/share/logstash/config/conf.d

These are my settings:
under /etc/default/logstash

LS_HOME="/usr/share/logstash"
LS_SETTINGS_DIR="/usr/share/logstash/config"
#LS_SETTINGS_DIR="/etc/logstash"
LS_PIDFILE="/var/run/logstash.pid"
LS_USER="logstash"
LS_GROUP="logstash"
LS_GC_LOG_FILE="/var/log/logstash/gc.log"
LS_OPEN_FILES="16384"
LS_NICE="19"
SERVICE_NAME="logstash"
SERVICE_DESCRIPTION="logstash"

Under /usr/share/logstash/config/logstash.yml

path.data: /var/lib/logstash
path.config: /usr/share/logstash/config/conf.d/*.conf
path.logs: /var/log/logstash

I have placed two .conf files under /usr/share/logstash/config/conf.d

Please help me as how to configure the logstash service which can refer to some *.conf files which will contain specific inputs,filters & outputs. The reason I want logstash to run as service is that there will be updates to the files being monitored (i.e. which we are interested to ingest data).

Thanks for taking a look at my question.

2

If you set variables in /etc/default/logstash then that script is sourced by /usr/share/logstash/bin/logstash, but they are not exported, so logstash itself never sees them. You need to export the variables.

3

Thanks @Badger for your response. This is what is present in /etc/init/logstash.conf

stop on runlevel [!2345]

respawn
umask 022
nice 19
limit nofile 16384 16384
chroot /
chdir /

#limit core <softlimit> <hardlimit>
#limit cpu <softlimit> <hardlimit>
#limit data <softlimit> <hardlimit>
#limit fsize <softlimit> <hardlimit>
#limit memlock <softlimit> <hardlimit>
#limit msgqueue <softlimit> <hardlimit>
#limit nice <softlimit> <hardlimit>
#limit nofile <softlimit> <hardlimit>
#limit nproc <softlimit> <hardlimit>
#limit rss <softlimit> <hardlimit>
#limit rtprio <softlimit> <hardlimit>
#limit sigpending <softlimit> <hardlimit>
#limit stack <softlimit> <hardlimit>


script
  # When loading default and sysconfig files, we use `set -a` to make
  # all variables automatically into environment variables.
  set -a
  [ -r "/etc/default/logstash" ] && . "/etc/default/logstash"
  [ -r "/etc/sysconfig/logstash" ] && . "/etc/sysconfig/logstash"
  set +a
 # exec chroot --userspec logstash:logstash / /usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash" >> /var/log/logstash-stdout.log 2>> /var/log/logstash-stderr.log
  exec chroot --userspec logstash:logstash / /usr/share/logstash/bin/logstash "--path.settings" "/usr/share/logstash/config" >> /var/log/logstash-stdout.log 2>> /var/log/logstash-stderr.log

end script

Sorry to ask basic question, how and where to export it?

4

Sorry, I was mistaken, I missed the 'set -a' when I read that script. That does the export for you. I do not know what the problem is.

5

oh ok. Thanks for trying to help me @Badger

6

I see you are dropping stdout/stderr into /var/log, not /var/log/logstash. Are there any clues in those?

7

ah, I also noticed just now. It displayed permission issue for logstash-plain.log. Fixed permission problem and now it is able to write and following is written in /var/log/logstash/logstash-plain.log

[2018-02-13T14:42:59,769][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<NameError: undefined local variable or method `dotfile' for #<AwesomePrint::Inspector:0x484eb628>
Did you mean?  @@dotfile>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/awesome_print-1.8.0/lib/awesome_print/inspector.rb:163:in `merge_custom_defaults!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/awesome_print-1.8.0/lib/awesome_print/inspector.rb:50:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/awesome_print-1.8.0/lib/awesome_print/core_ext/kernel.rb:9:in `ai'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-rubydebug-3.0.5/lib/logstash/codecs/rubydebug.rb:39:in `encode_default'", "org/jruby/RubyMethod.java:115:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-rubydebug-3.0.5/lib/logstash/codecs/rubydebug.rb:35:in `encode'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/base.rb:50:in `block in multi_encode'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/codecs/base.rb:50:in `multi_encode'", "/usr/share/logstash/logstash-core/lib/logstash/outputs/base.rb:90:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/single.rb:15:in `block in multi_receive'", "org/jruby/ext/thread/Mutex.java:148:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/single.rb:14:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:50:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:487:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:486:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:438:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:393:in `block in start_workers'"]}
8

I would remove the rubydebug codec from your output and see if you get a different error message.

9

It is able to ingest into elastic after commenting #stdout { codec => rubydebug }. But lot of parsing failures.

sample

[2018-02-13T14:55:22,305][WARN ][logstash.filters.csv ] Error parsing csv {:field=>"message", :source=>"\"INC

10

OK, so now you need to say what the input looks like, what your config looks like, and what you are hoping that config will do.

11

Thanks @Badger, I think i will look into those issues. But, is there any downside of commenting this line out? #stdout { codec => rubydebug }. Is it just used to display the parsed data in the local machine?

12

Yes, the rubydebug codec formats stdout for the logstash process. That is very useful for seeing how your messages are being parsed whilst debugging your configuration. I generally use an output like that while testing my filters. I don't send anything to elasticsearch until I know the filters are doing what I want.

13

Exactly. When I was testing also, I did not see any issues. The only difference was that i was using command line and option -f. So I did not feel that anything was wrong with rubydebug. Anyways, thanks for your time and help

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.