Elastic only receiving partial logstash events?


#1

Hi All,
I have a logstash forwarding syslog events from multiple applications on a single server:

input {

      tcp {
        port => 514
        type => syslog
      }
      udp {
        port => 514
        type => syslog
      }
    }
    
    filter {
      if [type] == "syslog" {
        grok {
          match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
          add_field => [ "received_at", "%{@timestamp}" ]
          add_field => [ "received_from", "%{host}" ]
        }
        syslog_pri { }
        date {
          match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
        }
      }
    }
    
    output {
      elasticsearch { host => localhost }
      stdout { codec => rubydebug }
    }

Elastic is only receiving logs from one application, and not another, however, I see both in the stdout. Any idea what may cause this?

Elasticsearch 1.5.2
Kibana 4.0.2
Logstash 1.5.0


(Mark Walkom) #2

Are the formats the same for both? As your grok pattern will be forced on them and if it doesn't match you should be seeing _grokparsefailures.


#3

The formats are just about the same.

It's actually weird, the grokparsefailure messages are showing, but the ones without failures aren't. I'm truly stumped


(system) #4