Don't save in ES logs that have in tag _grokparsefailure or _dateparsefailure

[Automation_Scripts]

Hi,

How can i filter the logs that are saved within ES in order to have only valid data that has no grok parse failure or date parse error? My approach is bellow

output {
  if ("_grokparsefailure" not in [tags]) or ("_dateparsefailure" not in [tags]) {
    elasticsearch {
      hosts => ["http://es01:9200","http://es02:9200","http://es03:9200"]
      # index => "%{[indexPrefix]}-logs-%{+dd.MM.YYYY}"
      index => "filebeat-kafka-logstash-%{+YYYY.MM.dd}"
      # index => "logstash-%{[type]}-%{+YYYY.MM.dd}"
      ecs_compatibility => disabled
    }
  }
}

[Badger]

@Automation_Scripts

if ("_grokparsefailure" not in [tags]) or ("_dateparsefailure" not in [tags]) {

I think where you wrote "or" you meant "and".

If the event has _dateparsefailure but not _grokparsefailure then the left will evaluate to true and the right will evaluate to false. When you "or" those you get true, so it will be indexed.

[Automation_Scripts]

ok, i solved it with a simple filter with drop. thank you!