Can't run Logstash as a Service

Simone1 · March 9, 2022, 11:13am

If i run logstash manually it works fine and sends all data in the right way.

But when i try to run it as a service it shows this error:

Cmd that i use: sudo systemctl start logstash.service
OS Ubuntu 20.04.1
Logstash version: 7.9.2

Pls help me, i need tofix it quickly

Badger · March 9, 2022, 6:13pm

You need to show us the logstash logs, possibly in /var/log/logstash.

Simone1 · March 9, 2022, 6:33pm

It doesn't generate any new log

leandrojmp · March 9, 2022, 6:34pm

You need to share your system logs as well.

Look at /var/log/messages or /var/log/syslog when you tried to start the logstash service to see why it was not able to start.

Simone1 · March 9, 2022, 6:37pm

It doesn't generate any new log

leandrojmp · March 9, 2022, 6:39pm

Should be something in /var/log/messages for Red Hat based distributions or /var/log/syslog for Debian based distributions.

Try to run sudo systemctl start logstash and check one of those files, without seeings logs of the system is not possible to know what is happening.

Also check the permissions to the /var/log/logstash folder to see if the logstash user has permissions to write on this folder.

Simone1 · March 10, 2022, 1:45pm

leandrojmp · March 10, 2022, 1:47pm

There is your error, logstash could not find java.

You will need to install java or set JAVA_HOME for the logstash processs.

You could also update to a newer version, starting with version 7.10 logstash now has a bundled java, but version 7.9 does not, so you need a java installation.

Simone1 · March 10, 2022, 1:50pm

i already have java,in:
java: /usr/bin/java /usr/share/java /usr/lib/jvm/java-11-oracle/bin/java

Where i set JAVA_HOME?

leandrojmp · March 10, 2022, 1:54pm

What the following command returns?

sudo -u logstash which java

This will tell you if the logstash user, the one that runs the service, can find the java binary in your system.

You could also add the variable into the file /etc/default/logstash or /etc/sysconfig/logstash depending on the distribution of your system.

You would need to add a line like this:

JAVA_HOME=/path/to/java/home

Simone1 · March 10, 2022, 1:58pm

If i lunch the command it doesn't give something in return

I use Ubuntu 20.04.1, so i need to modify /etc/default/logstash, right?

leandrojmp · March 10, 2022, 2:03pm

If the command does not return anything, than the logstash user can not find java in your system, that's the reason you cannot start the service.

You could add the variable to the /etc/default/logstash, but it would be better if you just upgrade the logstash to a new version as it already has a bundled jdk.

Simone1 · March 10, 2022, 2:23pm

I tried to start and nothing arrivied in Kibana, i looked at syslog and i found this:

leandrojmp · March 10, 2022, 2:28pm

Please, do not share screenshot of log messages, just copy the plain text and share it using the preformatted text button </>.

It is really hard to see error logs on screenshots and is not possible to search on them.

Your error is pretty clear, the logstash user does not have permission to write in the directory /var/log/logstash, so it can't create the log file.

Change the permission to the directory

sudo chown logstash:logstash /var/log/logstash

Simone1 · March 10, 2022, 2:38pm

i have the same error after using the command sudo chown

main ERROR RollingFileManager (/var/log/logstash/logstash-plain.log) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied) java.io.FileNotFoundException: /var/log/logstash/logstash-plain.log (Permission denied)

leandrojmp · March 10, 2022, 2:40pm

Did you tried to run logstash before as the root user? If you did it would create the file with the root user as the owner.

Just remove the file and start the service again.

sudo rm /var/log/logstash/logstash-plain.log

Simone1 · March 10, 2022, 2:46pm

I only run logstash manually before, as root

Now i have this:

[ERROR][org.logstash.execution.AbstractPipelineExt] Logstash failed to create queue.

[ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"java.nio.file.AccessDeniedException: /usr/share/logstash/queue_data/main/.lock", :backtrace=>["org.logstash.execution.AbstractPipelineExt.openQueue(AbstractPipelineExt.java:204)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$0$0$openQueue.call(AbstractPipelineExt$INVOKER$i$0$0$openQueue.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:831)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:45)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:834)"]}
Mar 10 15:42:41 lnx01bigdata logstash[140667]: warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):

leandrojmp · March 10, 2022, 2:55pm

That's the issue, when you run it as the root user, it creates a couple of files owned by the root user, mainly the log file and some files if you are using persistent queues, which seems your issue now.

You will need to remove or change the permission of those files created when you run it as a root, check the log, see what file it says that it does not have permission and delete or change the permissions to the logstash user.

Simone1 · March 10, 2022, 2:58pm

could be this?

java.nio.file.AccessDeniedException: /usr/share/logstash/queue_data/main/.lock

leandrojmp · March 10, 2022, 2:59pm

Yes, check the description of the error AccessDeniedException, this means that logstash could not access this file, so you need to change it permissions to the logstash user.