Duplicate entries after grok

Elastic Stack Logstash
#1

We have, for reasons, a massive error file that has many different message formats in it. Im just looking to initially get the line into the server, then I should be able to split up the message field after.

My config looks like this:

filter {
  if [type] == "errorfile" {
    grok {
      match => { "message" => "[%{SYSLOG5424SD:timestamp}] %{GREEDYDATA:message_remainder }"}
    }
    date {
      match => [ "timestamp", "dd-MMM-yyyy HH:mm:ss" ]
    }
    mutate  {
      replace => { "message" => "%{message_remainder}" }
    }
  }
}

This broadly works, but I now get a message field and and message_remainder field in kibana, sudo I added

  drop {
    remove_field => ["%{message_remainder}"]
  }

but it looks like it dropped the entire log message.

Baliclly what I am trying to achieve is to not have the timestamp in the message field because it already has its own field

#2

Indeed drop{} filter is for dropping event. To manipulate the event, always think mutate filter

So you could just add in your config

mutate {
  remove_field => ["message_remainder", "timestamp"]
}

Please note that you must simply use the field name, using the %{} would interpolate the field and use the field value. This is cool for event-driven configuration, but this is not your use case.

Just a note why you were misleaded, ALL plugins contains some basic manipulation of event like add_field,remove_field,... to be applied after sucessful handling of event so the drop filter also have them even if after dropping the event nothing can really be done.

Using this feature in the date filter directly and having a look to the overwrite config of grok you can simply do a config like this

 grok {
   match => { "message" => "[%{SYSLOG5424SD:timestamp}] %{GREEDYDATA:message }"}
   overwrite => [ "message" ]
}
date {
  match => [ "timestamp", "dd-MMM-yyyy HH:mm:ss" ]
  remove_field => ["timestamp"]
}

See, no need for mutate at all in simple cases :smiley:

#3

Exactly what I was looking for wiibaa. Thanks for the detailed response.

#4