When running Logstash in foreground it works, but logging file it always empty

Hi. Just this morning after I have come across an unusual problem with Logstash.
When it's running as a service 'ps aux | grep logstash' and 'systemctl status logstash' shows as its running but its not listening on any of the input ports nor is the /var/log/logstash file getting updated, but when I run it in the foreground the listeners work, remote system logs flow through LS --> ES and I can search on them fine but still the /var/log/logstash/logstash.log file isn't getting updated.

This was a working system and the last thing I was doing was creating a new index for Nginx access-logs which I have rolled back my changes to before. Even rebooted in case I had some kind of lock...

I am on Logstash 2.3.4 running on Linux.

Any ideas?

how are you starting logstash, by default logstash (as a service) starts as the logstash userid

if your starting as root and it is working then BINGO!

check your /etc/sysconfig/logstash and update it appropriately for your environment.

Logstash use to start with 'systemctl enable logstash' when I first installed it a few weeks back and 'systemctl start logstash'. Since then I just reboot box and services came up.

OS is based on Oracle Linux 7.2 / Centos 7.X.

I can run it in foreground with '/opt/logstash/bin/logstash -f /etc/logstash/conf.d/'..

Just did a test by deleting all my log files in /var/log/logstash and then restarted with 'systemctl restart logstash' and system re-created logstash.stdout and logstash.err but no logstash.log! .... strange.

yah that will be probably a simular issue,

the /var/log/logstash directory is probably owned by root (or not logstash), check this and see,.

I have an identical mirror image system running and it's working fine..
Permission for /var/log/logstash on this broken system is
drwxrwxr-x. 2 logstash root

Have also noticed that when it's run as a service its not listening on its ports configured in the input file, but run it in the foreground and alls well...

The issue you described sound very much like an owner ship problem. I can't put my figure around why it would not log, or bind to the port

Here is the scope of everything I would check, sort of a brain dump I hope it helps

service logstash status
if running ps -ef | grep logstash and verify the userid it is running as

If your udp port is less then 1025 then the process must be running as root

Check your /etc/syslog/logstash that it is owned by logstash and the LS_USER in that file is appropriate (I set it to root but it defaults to logstash)

If running as logstash
Check your /etc/logstash/conf.d and files in that directory are readable by that user
Also check /var/lib/logstash and as you did above /var/log/logstash and those files too

I generally turn off SELINUX on my system at build time. this could prevent your from writing or binding

You may want to add LS_OPTS='--debug' to your /etc/sysconfig/logstash file and there should be some debugging logs there.

Are you using the same Java version, don't know why but I thought I would ask.

If running as logstash sudo -iu logstash /opt/bin/logstash -f /etc/logstash/conf.d see if it works

after that I would go in to full on debugging

add strace -s0 -w /tmp/log to the script where it executes logstash see what the you might find there

or I would add set -x to the /etc/init.d/logstash (right under the #!/bin/bash and watch the script execute

Does your /var/log/messages or any other file mention anthing?

The last thought, try uninstalling and re-installing maybe something got corrupt.

Other then that without a config and some messages I am out of idea's of what would cause you a problem