Logstash doesn't appear to be reading config'd files


(Nathan) #1

Was running Logstash 1.5.0 on Debian 7 Wheezy.
It was running fine.
Upgraded to Debian 8 Jessie, and now it's running, but doesn't display data from the configured log files.
It looks like it's only reading authentication errors on the local system. A file I'd not even pointed it at.
I was seeing 2,000-3,000 logs per minute, now I'm not even getting 200-300/min.

I've tweaked and poked at the config, yet it still doesn't display anything from the assigned files.
It appears that lsof is saying the files are being read by Logstash.

init.d/logstash file does have the correct config directory and bin paths.
Logstash starts/stops/restarts correctly with no errors.

Any wisdom on where to start looking?

Input portion of the config (broken into three files: input, filter, output)

input {
    # syslog
    file {
             path => "/var/log/syslog" 
            exclude => "/var/log/syslog.*"
            type => "syslog" 
        }
    # all log
    file {
            path => "/var/log/*.log" 
            exclude => "/var/log/*.gz"
            type => "syslog"
            tags => ["swuc","other"] 
        }
    
    # Mail Logs
    file { 
            path => "/var/log/mail.log"
            path => "/var/log/mail.info"
            type => "mail"
            tags => [ "swuc","mail"]
    }
    # DMESG logs
    file {
            path => "/var/log/dmesg"
            type => "dmesg"
            tags => [ "swuc","dmesg"]
        }
    # ARLA Logs
    file {
            path => "/var/log/remote/arla.log"
            type => "syslog"
            tags => ["arla","syslog","remote"]
        }    
    # OKLA Logs
    file {
            path => "/var/log/remote/okla.log"
            type => "syslog"
            tags => ["okla","syslog","remote"]
        }    
    # SWUC Logs
    file {
            path => "/var/log/remote/swuc.log"
            type => "swuc"
            tags => ["swuc","syslog"]
        }    
    # SWUC-PA200 Logs
    file {
            path => "/var/log/remote/swuc-pa200.log"
            type => "paloalto"
            tags => ["swuc","paloalto","firewall"]
        }    

    # SWRG Logs
    file {
            path => "/var/log/remote/swrg.log"
            type => "syslog"
            tags => ["swrg","syslog","remote"]
        }
    # TXCO Logs
    file {
            path => "/var/log/remote/txco.log"
            type => "syslog"
            tags => ["txco","syslog","remote"]
        }
    # LSC Logs
    lumberjack {
        port => 5015
        ssl_certificate => ["/etc/ssl/certs/logstash-forwarder.crt"]
        ssl_key => ["/etc/ssl/private/logstash-forwarder.key"]
    }
    file {
            path => "/var/log/remote/lsc.log"
            type => "syslog"
            tags => ["lsc","syslog","remote"]
        }    
    # SWUC-Auditing Logs
    file {
            path => "/var/log/remote/swuc-audit.log"
            type => "swuc-audit"
        }    
    # SWUC-Wifi Logs
    file {
            path => "/var/log/remote/swuc-wifi.log"
            type => "swuc-wifi"
        }    
    # Apache Logs
    file {
            path => "/var/log/apache2/access.log"
            path => "/var/log/apache2/error.log"
            type => "Apache"
    }
    # Nginx Logs
    file {
            path => "/var/log/nginx/access.log"
            path => "/var/log/nginx/error.log"
            type => "Nginx"
    }
    # PaloAlto Logs (testing)
    tcp {
            port => "5000"
            type => "paloalto"
        }    
    # Windows Logs
    tcp { 
        port => 33444
        codec => json_lines
    }
}

(Magnus Bäck) #2

Increase logging verbosity with --verbose or --debug and see if it logs any access denied errors?


(Nathan) #3

Adding this to the /etc/init.d/logstash file doesn't change anything.
Running manually "/opt/logstash/bin/logstash agent -f -l --debug" tells me that I can't write the log file and doesn't run.

Looking at permissions details now.


(Nathan) #4

Permissions fixed.
Everything runs correctly when run manually.

Running it as a daemon results in it not reading the files, nor picking up the '--debug' flag in the init.d file.


(Magnus Bäck) #5

Running it as a daemon results in it not reading the files,

Presumably because it's running as another user?

nor picking up the '--debug' flag in the init.d file.

That's weird though. It should work. Note that the standard way of adding startup options is via /etc/default/logstash. I frequently modify the startup options that way.


(Nathan) #6

The only error I get when running it as a daemon is found in /var/log/logstash/logstash.err

WARNING: Default JAVA_OPTS will be overridden by the JAVA_OPTS defined in the environment. Environment JAVA_OPTS are -Djava.io.tmpdir=/opt/logstash

That field is commented out in /etc/default/logstash. It looks exactly like the one in init.d/logstash
I've not changed that field.
Not knowing enough about java, is that how it's supposed to be?
The line in the init.d/logstash file relating to that reads like this:

LS_JAVA_OPTS="-Djava.io.tmpdir=${LS_HOME}"

And LS_HOME looks like this:

LS_HOME=/opt/logstash


(Nathan) #7

Figured it out.
None of the other ELK pieces were using the LS_JAVA_OPTS field. All commented out.
Once I commented out that line in the logstash init.d file, it fired right up and is working as expected.

Thanks for the help!


(system) #8