File input not working - again

Elastic Stack Logstash
1

Hello folks,

I've got issues when tried to read syslog-ng files with file input. It simply doesn't work. Here is my config:

    input {
            file {
                    path => [ "/var/log/10-SWITCH/*" ]
                    sincedb_path => "/var/log/logstash/sincedb"
                    type => "syslog"
                    tags => [ "SWITCH" ]
            }
    }

    filter {
            if [type] == "syslog" {
                    grok {
                            patterns_dir => "/opt/logstash/patterns"
                            match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
                            add_field => [ "received_at", "%{@timestamp}" ]
                            add_field => [ "received_from", "%{host}" ]
                    }
                    syslog_pri { }
                    date {
                            match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
                    }
            }
    }

output {
    elasticsearch {
        host => localhost
        embedded => true
    }
    stdout { codec => rubydebug }
}

I tried:

  • Delete sincedb files, including those located at $HOME/.sincedb
  • Set sincedb_path to /dev/null
  • Set start_position to beginning
  • Set root:root owner/group to logs dir

None of above worked.

Use a socket (i.e.: input { tcp { port => 5514 } } } ) as input is an option, but I can't figure out a way to filter in origin the several different types of logs collected (firewalls, windows, linux process, wi-fi controllers, switches, routers and so on). I cannot do many changes to syslog-ng configuration, like listening port for exemple.

May some help with this issue?

Regards!

Fábio

2

Increase Logstash's logging verbosity with --verbose and inspect the logs. My hunch is that you don't have sufficient permissions to read the log files.

  • Set root:root owner/group to logs dir

How would this help? Are you running Logstash as root?

Use a socket (i.e.: input { tcp { port => 5514 } } } ) as input is an option, but I can't figure out a way to filter in origin the several different types of logs collected (firewalls, windows, linux process, wi-fi controllers, switches, routers and so on).

Why would this be more difficult if you're receiving the logs over the network rather than by reading local files? How are you currently distinguishing the different messages?

3

You got it Magnus!

Permission issue, I set read flag to group and group ownership to logstash in logs dir. Everything is working now!

Thanks!

Best regards!