Logstash giving error which is not clear

Elastic Stack Logstash
1

I am getting the following error in logstash-plain.log:

[2023-06-03T01:33:34,256][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-06-03T01:33:34,272][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.17.4", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [linux-x86_64]"}
[2023-06-03T01:33:34,276][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djdk.io.File.enableADS=true, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true]
[2023-06-03T01:33:34,285][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"fb_apache", :directory=>"/usr/share/logstash/modules/fb_apache/configuration"}
[2023-06-03T01:33:34,286][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"fb_apache", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x63d478a7 @directory="/usr/share/logstash/modules/fb_apache/configuration", @module_name="fb_apache", @kibana_version_parts=["6", "0", "0"]>}
[2023-06-03T01:33:34,288][DEBUG][logstash.modules.scaffold] Found module {:module_name=>"netflow", :directory=>"/usr/share/logstash/modules/netflow/configuration"}
[2023-06-03T01:33:34,288][DEBUG][logstash.plugins.registry] Adding plugin to the registry {:name=>"netflow", :type=>:modules, :class=>#<LogStash::Modules::Scaffold:0x47e0e23f @directory="/usr/share/logstash/modules/netflow/configuration", @module_name="netflow", @kibana_version_parts=["6", "0", "0"]>}
[2023-06-03T01:33:34,329][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

My logstash.yml looks like:

path.data: /var/lib/logstash
pipeline.id: main
log.level: debug
path.logs: /var/log/logstash

The logstash service status looks ok and running.
If I run my test.conf file located in /etc/logstash/conf.d/ with the command:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

I get the output sent to elastic nodes and I can see the documents in kibana.
But I still see the above errors in my logstash-plain.log file.
Please help.

2

Try this command and see if you get the same errors

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash

Also did you try

systemctl daemon-reload logstash

1 Like
3

This is an error on the start of logstash, you need to check in your systems logs, /var/log/messages or /var/log/syslog depending on your distro.

Check the system logs and look for any hint about any issue with logstash.

With the error you shared the service is not ok, it may be show as running/up when you run systemctl status logstash, but it is probably on a restart loop.

4

If doesn't help what Leandro suggest, try with Oracle JDK installation, not OpenJDK once I had similar issue and only Oracle JDK solve this. It might be related to java libs or paths which Oracle JDK set.

5

I tried the below command and got the same errors in my logstash-plain.log.

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf --path.settings /etc/logstash

I tried daemon-reload logstash but got invalid number of arguments error message:

[root@server1 logstash]# systemctl daemon-reload logstash
Invalid number of arguments
6

You were right.

  1. I looked under /var/log/messages and I found the message Your settings are invalid. Reason: Path "/var/lib/logstash/dead_letter_queue" must be a writable directory. It is not writable.
    So I ran chown logstash:logstash /var/lib/logstash/dead_letter_queue and restarted the logstash service.
  2. I checked the logstash-plain.log and got this error now:
    An unexpected error occurred! {:error=>java.nio.file.AccessDeniedException: /var/lib/logstash/.lock,
    To fix this I deleted the .lock file from /var/lib/logstash and restarted the logstash service.

After these I don't see any errors in my logstash-plain.log and now it's a good weekend :slight_smile:
Thank you for your help.

2 Likes
7

Apologies command is, glad you got it working

8

I tried that too, and then restarted the logstash service but it didn't fix the issue.
Thank you for helping me on this.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.