# Duplicate entries after grok

**URL:** <https://discuss.elastic.co/t/duplicate-entries-after-grok/41644>\
**Category:** Logstash\
**Created:** [February 12, 2016, 5:01pm UTC](https://discuss.elastic.co/t/duplicate-entries-after-grok/41644 "2016-02-12T17:01:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![moncky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moncky/32/6320_2.png) [@moncky](https://discuss.elastic.co/u/moncky)\
**Post date:** [February 12, 2016, 5:01pm UTC](https://discuss.elastic.co/t/duplicate-entries-after-grok/41644/1 "2016-02-12T17:01:56Z")

</div>

We have, for reasons, a massive error file that has many different message formats in it. Im just looking to initially get the line into the server, then I should be able to split up the message field after.

My config looks like this:

```
filter {
  if [type] == "errorfile" {
    grok {
      match => { "message" => "[%{SYSLOG5424SD:timestamp}] %{GREEDYDATA:message_remainder }"}
    }
    date {
      match => ["timestamp", "dd-MMM-yyyy HH:mm:ss"]
    }
    mutate {
      replace => { "message" => "%{message_remainder}" }
    }
  }
}

```

This broadly works, but I now get a message field and and message\_remainder field in kibana, sudo I added

```
  drop {
    remove_field => ["%{message_remainder}"]
  }

```

but it looks like it dropped the entire log message.

Baliclly what I am trying to achieve is to not have the timestamp in the message field because it already has its own field

---

<div class="post-metadata">

**Author:** ![wiibaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wiibaa/32/44931_2.png) [@wiibaa](https://discuss.elastic.co/u/wiibaa)\
**Post date:** [February 13, 2016, 7:45am UTC](https://discuss.elastic.co/t/duplicate-entries-after-grok/41644/2 "2016-02-13T07:45:36Z")

</div>

Indeed drop{} filter is for dropping event. To manipulate the event, always think **mutate** filter

So you could just add in your config

```
mutate {
  remove_field => ["message_remainder", "timestamp"]
}

```

Please note that you must simply use the field name, using the **%{}** would interpolate the field and use the field value. This is cool for event-driven configuration, but this is not your use case.

Just a note why you were misleaded, ALL plugins contains some basic manipulation of event like add\_field,remove\_field,... to be **applied after sucessful handling of event** so the drop filter also have them even if after dropping the event nothing can really be done.

Using this feature in the date filter directly and having a look to the **[overwrite](https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-overwrite)** config of grok you can simply do a config like this

```
 grok {
   match => { "message" => "[%{SYSLOG5424SD:timestamp}] %{GREEDYDATA:message }"}
   overwrite => ["message"]
}
date {
  match => ["timestamp", "dd-MMM-yyyy HH:mm:ss"]
  remove_field => ["timestamp"]
}

```

See, no need for mutate at all in simple cases 😃

---

<div class="post-metadata">

**Author:** ![moncky](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moncky/32/6320_2.png) [@moncky](https://discuss.elastic.co/u/moncky)\
**Post date:** [February 15, 2016, 12:21pm UTC](https://discuss.elastic.co/t/duplicate-entries-after-grok/41644/3 "2016-02-15T12:21:14Z")

</div>

Exactly what I was looking for wiibaa. Thanks for the detailed response.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:11am UTC](https://discuss.elastic.co/t/duplicate-entries-after-grok/41644/4 "2017-07-06T05:11:30Z")

</div>