When running Logstash in foreground it works, but logging file it always empty

Elastic Stack Logstash
1

Hi. Just this morning after I have come across an unusual problem with Logstash.
When it's running as a service 'ps aux | grep logstash' and 'systemctl status logstash' shows as its running but its not listening on any of the input ports nor is the /var/log/logstash file getting updated, but when I run it in the foreground the listeners work, remote system logs flow through LS --> ES and I can search on them fine but still the /var/log/logstash/logstash.log file isn't getting updated.

This was a working system and the last thing I was doing was creating a new index for Nginx access-logs which I have rolled back my changes to before. Even rebooted in case I had some kind of lock...

I am on Logstash 2.3.4 running on Linux.

Any ideas?

2

how are you starting logstash, by default logstash (as a service) starts as the logstash userid

if your starting as root and it is working then BINGO!

check your /etc/sysconfig/logstash and update it appropriately for your environment.

3

Hi.
Logstash use to start with 'systemctl enable logstash' when I first installed it a few weeks back and 'systemctl start logstash'. Since then I just reboot box and services came up.

OS is based on Oracle Linux 7.2 / Centos 7.X.

I can run it in foreground with '/opt/logstash/bin/logstash -f /etc/logstash/conf.d/'..

Just did a test by deleting all my log files in /var/log/logstash and then restarted with 'systemctl restart logstash' and system re-created logstash.stdout and logstash.err but no logstash.log! .... strange.

4

yah that will be probably a simular issue,

the /var/log/logstash directory is probably owned by root (or not logstash), check this and see,.

5

I have an identical mirror image system running and it's working fine..
Anyway.
Permission for /var/log/logstash on this broken system is
drwxrwxr-x. 2 logstash root

Have also noticed that when it's run as a service its not listening on its ports configured in the input file, but run it in the foreground and alls well...

6

The issue you described sound very much like an owner ship problem. I can't put my figure around why it would not log, or bind to the port

Here is the scope of everything I would check, sort of a brain dump I hope it helps

service logstash status
if running ps -ef | grep logstash and verify the userid it is running as

If your udp port is less then 1025 then the process must be running as root

Check your /etc/syslog/logstash that it is owned by logstash and the LS_USER in that file is appropriate (I set it to root but it defaults to logstash)

If running as logstash
Check your /etc/logstash/conf.d and files in that directory are readable by that user
Also check /var/lib/logstash and as you did above /var/log/logstash and those files too

I generally turn off SELINUX on my system at build time. this could prevent your from writing or binding

You may want to add LS_OPTS='--debug' to your /etc/sysconfig/logstash file and there should be some debugging logs there.

Are you using the same Java version, don't know why but I thought I would ask.

If running as logstash sudo -iu logstash /opt/bin/logstash -f /etc/logstash/conf.d see if it works

after that I would go in to full on debugging

add strace -s0 -w /tmp/log to the script where it executes logstash see what the you might find there

or I would add set -x to the /etc/init.d/logstash (right under the #!/bin/bash and watch the script execute

Does your /var/log/messages or any other file mention anthing?

The last thought, try uninstalling and re-installing maybe something got corrupt.

Other then that without a config and some messages I am out of idea's of what would cause you a problem