Mutate only matching

Elastic Stack Logstash
1

Hi,
I'd like to "extract" some fields from a specific kind of log and convert them to the right type.
I can do that when all logs pass the match but in this case the logs I'm getting have different formats.
So I would like to let the ones not matching go through ( without failing, like if there was no "filter" at all) and only mutate the matching ones.
Now the ones not matching give error and don't go through.

filter {
    grok {
       match => { "message" => "%{TIMESTAMP_ISO8601:ih_timestamp} %{WORD:name} (?<component>\w+\[\d+\]\:) *"}
    }
    date {
      match => [ "ih_timestamp", "ISO8601"]
      target => "@timestamp"
      remove_field => [ "ih_timestamp" ]
    }

    kv {
       field_split => ", "
    }

    mutate {
       convert => { "duration" => "integer" }
       convert => { "bytes" => "integer" }
    }
  
}

Thanks for any help.

2

Grok failures never cause events to get dropped (unless you explicitly configure Logstash that way). The only consequence of a grok failure is that the event is tagged _grokparsefailure, but that behavior can be influenced with the tag_on_failure option.

3

Thanks for replying,
I added:
tag_on_failure => []

So am I right to assume that if the matching pattern is not met then the rest of the pipeline is skipped (kv, mutate) ?

4

So am I right to assume that if the matching pattern is not met then the rest of the pipeline is skipped

I repeat: The only consequence of a grok failure is that the event is tagged _grokparsefailure, but that behavior can be influenced with the tag_on_failure option. The rest of the pipeline is not skipped.

5

Thanks again,
and what if I wanted to skip kv, and mutate when logs are not matching in grok ?

6

There's an example here:
https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html#conditionals

1 Like