How to handle _grokparsefailure with multiple inputs

Elastic Stack Logstash
1

Hello, I have logstash with two inputs: beats and gelf. Logs that come from beats are syslog messages, logs from gelf are docker logs.
I created a filter to match multiline logs from docker that works perfect, however I'm getting a _grokparsefailure for the beats entries, although they are properly parsed.

What am I doing wrong?

This is my logstash config:

input {
  beats {
    port => 5044
  }
}

input {
  gelf {}
}

filter{
  grok {
    match => { "image_name" => "[^/]/(?<application>[^:]+)" }
    tag_on_failure => ["NOT_APP"]
  }
  if [application] == "web-modeler" {
    multiline {
      pattern => "^%{TIME}"
      negate => true
      what => "previous"
      source => "message"
      stream_identity => "%{host}.%{container_id}"
    }
    grok {
      match => [ "message", "%{TIME:time} \[(?<thread_id>(?:[^\]]+))\] %{LOGLEVEL:severity}%{SPACE}%{JAVACLASS:logger_class} - %{GREEDYDATA:message}" ]
      overwrite => [ "message" ]
    }
  }

  grok {
    match => [ "message", "%{SYSLOGLINE}" ]
    overwrite => [ "message" ]
  }
  grok {
    match => [ "message", "%{MONTH} %{MONTHDAY} %{TIME} %{HOSTNAME} %{SYSLOGPROG}" ]
    overwrite => [ "message" ]
  }
}

output {
  if [application] == "web-modeler" {
    elasticsearch {
      hosts => ["elastic-1.local", "elastic-2.local", "elastic-0.local"]
      index => "web-modeler-%{+YYYY.MM.dd}"
    }
  } else {
    elasticsearch {
      hosts => ["elastic-1.local", "elastic-2.local", "elastic-0.local"]
      index => "common-%{+YYYY.MM.dd}"
    }
  }
}

The goal is to have different treatment for beats msgs than from gelf msgs, then to send gelf logs to one index and beats to another. However, I'm getting the parse error in the first statement (beats msgs don't have "image_name" in their tags)

Thanks!

2

I suggest you set the type in the input and add conditionals to

input {
  beats {
    port => 5044
    type => "syslog"
  }
}

input {
  gelf {
    type => "gelf"
  }
}

filter {
  if [type] == "syslog" {
    ...
  }
}
3

This actually helped a lot, thank you very much @magnusbaeck!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.