Don't save in ES logs that have in tag _grokparsefailure or _dateparsefailure

Elastic Stack Logstash
1

Hi,

How can i filter the logs that are saved within ES in order to have only valid data that has no grok parse failure or date parse error? My approach is bellow

output {
  if ("_grokparsefailure" not in [tags]) or ("_dateparsefailure" not in [tags]) {
    elasticsearch {
      hosts => ["http://es01:9200","http://es02:9200","http://es03:9200"]
      # index => "%{[indexPrefix]}-logs-%{+dd.MM.YYYY}"
      index => "filebeat-kafka-logstash-%{+YYYY.MM.dd}"
      # index => "logstash-%{[type]}-%{+YYYY.MM.dd}"
      ecs_compatibility => disabled
    }
  }
}
2

I think where you wrote "or" you meant "and".

If the event has _dateparsefailure but not _grokparsefailure then the left will evaluate to true and the right will evaluate to false. When you "or" those you get true, so it will be indexed.

3

ok, i solved it with a simple filter with drop. thank you!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.