Logstash running as service is not sending logs

[alesabat]

Hi there!

I have a rhel ppc64le with tomcat. Wanted to send logs to ELK cluster (filebeat discarted because has no ppc version, so I choose logstash).

If I run logstash from CLI, it works fine and I can discover logs in KIbana. But if I start logstash service, stop sending logs. Theese are commands issued:

/bin/logstash -f /etc/logstash/conf.d/myconf.conf -path.settings /etc/logstash (works fine).

systemctl stasrt logstash (service starts OK, no error logs, but not sending logs to ELK).

I checked /etc/logstash/logstash.yml, /etc/logstasg/pipelines.yml, etc... and everythings looks correctly.

I will very appreciatte any help.

Regards!

[Badger]

Try

systemctl show logstash.service | grep ExecStart

Then run the command that that shows.

[alesabat]

This is the output:

ExecStart={ path=/usr/share/logstash/bin/logstash ; argv=/usr/share/logstash/bin/logstash --path.settings /etc/logstash ; ignore_errors=no ; start_time=[Wed 2019-11-20 15:51:15 -03] ; stop_time=[n/a] ; pid=21171 ; code=(null) ; status=0/0 }

[Badger]

@alesabat

/usr/share/logstash/bin/logstash --path.settings /etc/logstash

What is logged when you execute that? (You will need to stop the service before doing so.)

[alesabat]

Okay, I did it (I did run with root user and logstash user, getting same behavior).

It's weird but is not logging anything (I'm checking /var/log/logstash/logstash-plain.log and /var/log/messages). No logs are sending to Kibana, but process is running.

[rugenl (Len Rugen)]

When you ran from the command line as root, files were probably created owned by root. When you run from systemctl, it will probably run with user logstash and won't have permission to the files it needs to run. The pid file is the most notorious.

So, check file ownership

[alesabat]

Hi, thanks for your answer. I've checked this out, this is what I got:

I have this lines from /etc/logstash/startup.options:

\# pidfiles aren't used the same way for upstart and systemd; this is for sysv users.
LS_PIDFILE=/var/run/logstash.pid

/var/run is a symbolic link which points to /run.

ls /run/logstash.pid file doesn't exists (logstash service status is running). Shoul I create it?

Theese are /run permissions:

drwxr-xr-x. 32 root root 980 Nov 21 10:57 /run

I added write permissions to others, restarted logstash but problem is still there.