Logstash starts but doesn’t do anything

Elastic Stack Logstash
1

Im trying to get some data in my logstash but my service doesn't seem to work properly. When I check my logstash status I get the following:

● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2021-12-13 11:29:25 UTC; 6s ago
Main PID: 1531953 (java)
Tasks: 18 (limit: 101101)
Memory: 521.1M
CGroup: /system.slice/logstash.service
└─1531953 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:>

Dec 13 11:29:25 TCSLSPRDWEAP2 systemd[1]: Started logstash.
Dec 13 11:29:25 TCSLSPRDWEAP2 logstash[1531953]: Using bundled JDK: /usr/share/logstash/jdk
Dec 13 11:29:25 TCSLSPRDWEAP2 logstash[1531953]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version>
~

Also logs show the following error

[2021-12-13T11:31:08,573][INFO ][logstash.runner] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2021-12-13T11:31:08,581][INFO ][logstash.runner] Starting Logstash {"logstash.version"=>"7.16.1", "jruby.version"=>"jruby 9.
2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2021-12-13T11:31:08,881][FATAL][logstash.runner] An unexpected error occurred! {:error=>java.nio.file.AccessDeniedException:
/var/lib/logstash/.lock, :backtrace=>["sun.nio.fs.UnixException.translateToIOException(sun/nio/fs/UnixException.java:90)", "sun.nio.fs
.UnixException.rethrowAsIOException(sun/nio/fs/UnixException.java:111)", "sun.nio.fs.UnixException.rethrowAsIOException(sun/nio/fs/Unix
Exception.java:116)", "sun.nio.fs.UnixFileSystemProvider.newFileChannel(sun/nio/fs/UnixFileSystemProvider.java:182)", "java.nio.channel
s.FileChannel.open(java/nio/channels/FileChannel.java:292)", "java.nio.channels.FileChannel.open(java/nio/channels/FileChannel.java:345
)", "org.logstash.FileLockFactory.obtainLock(org/logstash/FileLockFactory.java:89)", "jdk.internal.reflect.NativeMethodAccessorImpl.inv
oke0(Native Method)", "jdk.internal.reflect.NativeMethodAccessorImpl.invoke(jdk/internal/reflect/NativeMethodAccessorImpl.java:62)", "j
dk.internal.reflect.DelegatingMethodAccessorImpl.invoke(jdk/internal/reflect/DelegatingMethodAccessorImpl.java:43)", "java.lang.reflect
.Method.invoke(java/lang/reflect/Method.java:566)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javas
upport/JavaMethod.java:471)", "org.jruby.javasupport.JavaMethod.invokeStaticDirect(org/jruby/javasupport/JavaMethod.java:373)", "RUBY.e
xecute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:421)", "usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clam
p_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68)", "
usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:291)", "usr.sha
re.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/j
ruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133)", "usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bo
otstrap/environment.rb:93)", "java.lang.invoke.MethodHandle.invokeWithArguments(java/lang/invoke/MethodHandle.java:710)", "org.jruby.Ru
by.runScript(org/jruby/Ruby.java:1205)", "org.jruby.Ruby.runNormally(org/jruby/Ruby.java:1128)", "org.jruby.Ruby.runNormally(org/jruby/
Ruby.java:1146)", "org.jruby.Ruby.runFromMain(org/jruby/Ruby.java:958)", "org.logstash.Logstash.run(org/logstash/Logstash.java:161)", "
org.logstash.Logstash.main(org/logstash/Logstash.java:71)"]}
[2021-12-13T11:31:08,884][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
at usr.share.logstash.lib.bootstrap.environment.(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
~

Not able to figure out the issue.

2

Your service is exiting as your log shows:

[2021-12-13T11:31:08,884][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit

The issue is also in your logs:

[2021-12-13T11:31:08,881][FATAL][logstash.runner] An unexpected error occurred! {:error=>java.nio.file.AccessDeniedException:
/var/lib/logstash/.lock,

The service can't create the lock file .lock in the path /var/lib/logstash.

Did you run logstash as a root before and now are you starting to run it as a service, using the logstash user? Do you have other instance running in the same server? How did you install logstash?

If you do not have other instance running in the same server, try to remove the file /var/lib/logstash/.lock and start the service again.

3

Thank you @leandrojmp

Did you run Logstash as a root before and now are you starting to run it as a service, using the Logstash user? -- I am running the Logstash as Root user only.
Do you have other instance running in the same server? -- NO
How did you install logstash? -- Ref Link: Installing Logstash | Logstash Reference [7.16] | Elastic -- This is the ref link I followed to install logstash

If you do not have other instance running in the same server, try to remove the file /var/lib/logstash/.lock and start the service again. -- how to remove this file? will there be no issues if I remove it. How to create it again ?

4

The .lock file is created by logstash when it starts, you do not create it manually.

If you installed Logstash using a package manager like yum or apt, then it will create a Logstash service that you start using systemctl start logstash, when you start the logstash service it runs using the logstash user, not root. You should not run logstash as root for security reasons.

If the service can't access the file /var/lib/logstash/.lock, this probably means that the logstash user does not have permission to this file, this normally happens when logstash was run before as a root user, with something like sudo /usr/share/logstash/bin/logstash.

As I said, try to remove the .lock file and start logstash again, use sudo rm /var/lib/logstash/.lock.

5

Thank you so much @leandrojmp

Issue is resolved.

I have removed the .lock file using sudo rm /var/lib/logstash/.lock .
And started the logstash. Now data is being ingested .

But getting error "main ERROR Null object returned for RollingFile in Appenders", as soon as start the logstash, but later data is being ingested. Will there be any issue?

● logstash.service - logstash
Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
Active: active (running) since Tue 2021-12-14 14:00:08 UTC; 16s ago
Main PID: 1871689 (java)
Tasks: 20 (limit: 101101)
Memory: 669.7M
CGroup: /system.slice/logstash.service
└─1871689 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:>

Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: at org.jruby.Ruby.runNormally(Ruby.java:1128)
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: at org.jruby.Ruby.runNormally(Ruby.java:1146)
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: at org.jruby.Ruby.runFromMain(Ruby.java:958)
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: at org.logstash.Logstash.run(Logstash.java:161)
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: at org.logstash.Logstash.main(Logstash.java:71)
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: 2021-12-14 14:00:24,157 main ERROR Null object returned for RollingFile in Appenders.
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: 2021-12-14 14:00:24,158 main ERROR Unable to locate appender "deprecation_plain_rolli>
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: 2021-12-14 14:00:24,158 main ERROR Unable to locate appender "deprecation_plain_rolli>
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: [2021-12-14T14:00:24,178][INFO ][logstash.runner ] Log4j configuration path >
Dec 14 14:00:24 TCSLSPRDWEAP2 logstash[1871689]: [2021-12-14T14:00:24,186][INFO ][logstash.runner ] Starting Logstash {"logst>

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.