Logstash service is active, enabled but netstat output shows port not listening

jayadevp · December 6, 2023, 6:07am

If i run the command to manually run logstash
" sudo /usr/share/logstash/bin/logstash -f "/etc/logstash/conf.d/fortigate.conf" --config.reload.automatic"
im able to see the output and netstat also shows port listening in "5144"
Myconfig file is as below,
input {
udp {
host => "x.x.x.x"
port => 5144
}
}
output {
# stdout {}
elasticsearch {
hosts => ["https://x.x.x.x"]
index => "firewall-%{+YYYY.MM.dd}"
user => "elastic"
password => " "
ssl => true
cacert => "/etc/logstash/certs/http_ca.crt"
}
}`

But if i stop the command the output stops and the port is also not listening which is specified to "5144"

Rios · December 6, 2023, 7:32am

Welcome to the community.

Have you enable the UDP 5144 port on the local firewall?

jayadevp · December 6, 2023, 7:49am

Are you referring to the builtin firewall?
I'm using Ubuntu22.04 and ufw is inactive

Rios · December 6, 2023, 8:45am

Can you see data with stdout?
If you do and when you stop the command, then you can run it in the service mode: systemctl start logstash or as the background process with & at the end of your command.

jayadevp · December 6, 2023, 9:11am

Can you see data with stdout?
Yes i can see it if i uncomment the stdout and run the command manually

you can run it in the service mode:
Yes.. The service is up and running. Still i dont see the port listening in 5144

Rios · December 6, 2023, 9:25am

Check permissions for /etc/logstash/conf.d/fortigate.conf. You are running LS as root from the cmd line. The LS service is using logstash:logstash account.

The /usr/lib/systemd/system/logstash.service or /etc/systemd/system/logstash.service file contains starting params.

jayadevp · December 6, 2023, 10:10am

Below is my logstash.service file

Which all should i change the permissions for?

Rios · December 6, 2023, 10:46am

It's already written.

leandrojmp · December 6, 2023, 1:22pm

Please share the netstat command you are running and the output of it.

Also, restart the logstash service to get fresh logs and share the logs as plain text, not as screenshots.

jayadevp · December 6, 2023, 6:31pm

Output of netstat

:~$ sudo netstat -tupln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 172.29.21.10:5601       0.0.0.0:*               LISTEN      849/node
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      816/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      910/sshd: /usr/sbin
tcp6       0      0 :::22                   :::*                    LISTEN      910/sshd: /usr/sbin
tcp6       0      0 :::9200                 :::*                    LISTEN      83956/java
tcp6       0      0 172.29.21.10:9300       :::*                    LISTEN      83956/java
udp        0      0 127.0.0.53:53           0.0.0.0:*                           816/systemd-resolve

Permissions of config file

 sudo ls -l /etc//logstash/conf.d/fortigate.conf
-rw-r--r-- 1 root root 844 Dec  6 05:12 /etc//logstash/conf.d/fortigate.conf

Logstash Logs:

Dec  6 17:39:30 srnvlsrv04 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Dec  6 17:39:30 srnvlsrv04 systemd[1]: logstash.service: Failed with result 'exit-code'.
Dec  6 17:39:30 srnvlsrv04 systemd[1]: logstash.service: Consumed 40.912s CPU time.
Dec  6 17:39:30 srnvlsrv04 systemd[1]: logstash.service: Scheduled restart job, restart counter is at 4.
Dec  6 17:39:30 srnvlsrv04 systemd[1]: Stopped logstash.
Dec  6 17:39:30 srnvlsrv04 systemd[1]: logstash.service: Consumed 40.912s CPU time.
Dec  6 17:39:30 srnvlsrv04 systemd[1]: Started logstash.
Dec  6 17:39:30 srnvlsrv04 logstash[628765]: Using bundled JDK: /usr/share/logstash/jdk
Dec  6 17:39:40 srnvlsrv04 logstash[628765]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
Dec  6 17:39:41 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:41,037][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
Dec  6 17:39:41 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:41,042][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.11.1", "jruby.version"=>"jruby 9.4.2.0 (3.1.0) 2023-03-08 90d2913fda OpenJDK 64-Bit Server VM 17.0.9+9 on 17.0.9+9 +indy +jit [x86_64-linux]"}
Dec  6 17:39:41 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:41,045][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
Dec  6 17:39:41 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:41,914][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:42,307][INFO ][org.reflections.Reflections] Reflections took 84 ms to scan 1 urls, producing 132 keys and 464 values
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:42,598][ERROR][logstash.outputs.elasticsearch] Invalid setting for elasticsearch output plugin:
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:   output {
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:     elasticsearch {
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:       # This setting must be a path
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:       # File does not exist or cannot be opened /etc/logstash/certs/http_ca.crt
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:       cacert => "/etc/logstash/certs/http_ca.crt"
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:       ...
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:     }
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:   }
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:42,632][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1318)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:452)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:561)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:516)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:110)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:42,648][INFO ][logstash.runner          ] Logstash shut down.
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: [2023-12-06T17:39:42,651][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: org.jruby.exceptions.SystemExit: (SystemExit) exit
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: #011at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:795) ~[jruby.jar:?]
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: #011at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:758) ~[jruby.jar:?]
Dec  6 17:39:42 srnvlsrv04 logstash[628765]: #011at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
Dec  6 17:39:42 srnvlsrv04 systemd[1]: logstash.service: Main process exited, code=exited, status=1/FAILURE
Dec  6 17:39:42 srnvlsrv04 systemd[1]: logstash.service: Failed with result 'exit-code'.
Dec  6 17:39:42 srnvlsrv04 systemd[1]: logstash.service: Consumed 40.324s CPU time.
Dec  6 17:39:43 srnvlsrv04 systemd[1]: logstash.service: Scheduled restart job, restart counter is at 5.
Dec  6 17:39:43 srnvlsrv04 systemd[1]: Stopped logstash.
Dec  6 17:39:43 srnvlsrv04 systemd[1]: logstash.service: Consumed 40.324s CPU time.
Dec  6 17:39:43 srnvlsrv04 systemd[1]: Started logstash.
Dec  6 17:39:43 srnvlsrv04 logstash[628833]: Using bundled JDK: /usr/share/logstash/jdk
^C

stephenb · December 6, 2023, 6:42pm

jayadevp:

[2023-12-06T17:39:42,598][ERROR][logstash.outputs.elasticsearch] Invalid setting for elasticsearch output plugin:
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:   output {
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:     elasticsearch {
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:       # This setting must be a path
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:       # File does not exist or cannot be opened /etc/logstash/certs/http_ca.crt
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:       cacert => "/etc/logstash/certs/http_ca.crt"
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:       ...
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:     }
Dec  6 17:39:42 srnvlsrv04 logstash[628765]:   }

Thats a problem...

jayadevp · December 6, 2023, 6:58pm

Hi Stephen,
I have also changed the ownership and given 777 permissions to the file for testing, mentioned as below, but still i'm getting the same error mentioned in the above post

@srnvlsrv04:~$ sudo ls -l /etc/logstash/certs/http_ca.crt
-rwxrwxrwx 1 logstash logstash 1915 Dec  4 13:45 /etc/logstash/certs/http_ca.crt

leandrojmp · December 6, 2023, 7:29pm

What is the result of the following command:

sudo -u logstash ls -l /etc/logstash/certs/http_ca.crt

Does the Logstash user has permissions to the /etc/logstash/certs path as well?

In your case, the Logstash service is not active, it is probably on a restart loop because of some permission error, so since it is not starting, it will never bind to the specific udp port.

You need to find what is the permission issue and fix that.

jayadevp · December 6, 2023, 7:36pm

Hi Leandro,
Looks like logstash is not able access the certs folder and the http_ca.crt file even after changing the permissions and ownership

:~$ sudo -u logstash ls -l /etc/logstash/certs/
ls: cannot access '/etc/logstash/certs/http_ca.crt': Permission denied
total 0
-????????? ? ? ? ?            ? http_ca.crt
@srnvlsrv04:~$ sudo -u logstash ls -l /etc/logstash/certs/http_ca.crt
ls: cannot access '/etc/logstash/certs/http_ca.crt': Permission denied
@srnvlsrv04:~$
@srnvlsrv04:~$
@srnvlsrv04:~$
@srnvlsrv04:~$ sudo ls -l /etc/logstash/certs/
total 4
-rwxrwxrwx 1 logstash logstash 1915 Dec  4 13:45 http_ca.crt

Please guide on how to fix it as i'm just a beginner with Linux.

leandrojmp · December 6, 2023, 7:38pm

What are the permissions for the /etc/logstash/certs path?

Run sudo ls -l /etc/logstash and share the result.

jayadevp · December 6, 2023, 7:40pm

@srnvlsrv04:~$ sudo ls -l /etc/logstash/certs/
total 4
-rwxrwxrwx 1 logstash logstash 1915 Dec  4 13:45 http_ca.crt

@srnvlsrv04:~$ sudo ls -l /etc/logstash/
total 48
drw-rw---- 2 logstash logstash  4096 Dec  4 13:45 certs
drwxr-xr-x 2 root     root      4096 Dec  6 19:19 conf.d
-rw-r--r-- 1 root     root      1833 Nov 11 08:20 jvm.options
-rw-r--r-- 1 root     root      7437 Nov 11 08:20 log4j2.properties
-rw-r--r-- 1 root     root       342 Nov 11 08:20 logstash-sample.conf
-rw-r--r-- 1 root     root     15476 Dec  6 17:38 logstash.yml
-rw-r--r-- 1 root     root       339 Dec  6 05:32 pipelines.yml
-rw------- 1 root     root      1696 Nov 11 08:20 startup.options

leandrojmp · December 6, 2023, 7:45pm

These permissions are not correct, a directory need to have an execute permission x and the certs directory does not have, so even if the owner is the logstash user, it will not be able to enter the directory to read the file.

Also, the owner of directories inside /etc should be root in most of the cases, not a normal user.

Try to run the following to fix your permissions.

sudo chmod 755 /etc/logstash/certs
sudo chown root:root /etc/logstash/certs

jayadevp · December 6, 2023, 7:54pm

Hi leandro,

Thank you very much for your assistance. Your solution worked

system · January 3, 2024, 7:55pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.