# Dropping / Mutating first part of log event

**URL:** <https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063>\
**Category:** Logstash\
**Created:** [April 1, 2016, 12:48pm UTC](https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063 "2016-04-01T12:48:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![joserose](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@joserose](https://discuss.elastic.co/u/joserose)\
**Post date:** [April 1, 2016, 12:48pm UTC](https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063/1 "2016-04-01T12:48:26Z")

</div>

Hi there

I'm attempting to parse the following log entry:

A, [2016-01-27T11:49:29.702997 #90] ANY -- : 2016-01-27 11:49:29 +0000 severity=INFO, {"method":"GET","path":"/404","format":"_/_","controller":"errors","action":"routing","status":200,"duration":4.45,"view":2.34,"db":0.0,"current\_user":null,"current\_user\_id":null,"current\_user\_something":null,"request\_ip":"127.0.0.1","@timestamp":"2016-01-27T11:49:29.702Z","@version":"1","message":"[200] GET /404 (errors#routing)"}

so I only get use part the includes

{"method": ......... routing)"}

(i.e. drop on the floor " A, [2016-01-27T11:49:29.702997 #90] ANY -- : 2016-01-27 11:49:29 +0000 severity=INFO, ")

which I _believe_ will be filtered correctly using the default json filter. Can anyone help me break this out using grok (and/or/mutate/kv filtering)? so I can add this to my ES indices?

Thanks very much in advance!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 3, 2016, 7:42pm UTC](https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063/2 "2016-04-03T19:42:41Z")

</div>

You're on the right track in using a grok filter to remove the boring parts and using a json filter to parse the remainder. Try using [http://grokconstructor.appspot.com/](http://grokconstructor.appspot.com/) to construct your grok expression. Its incremental construction feature should be helpful.

---

<div class="post-metadata">

**Author:** ![mick66](https://avatars.discourse-cdn.com/v4/letter/m/d2c977/32.png) [@mick66](https://discuss.elastic.co/u/mick66)\
**Post date:** [April 4, 2016, 11:27am UTC](https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063/3 "2016-04-04T11:27:40Z")

</div>

Try this:

```
filter {
    grok {
        match => ["message", "(?<fields_to_keep>{%{GREEDYDATA}})"]
    }

    json {
        source => "fields_to_keep"
    }

    mutate {
        remove_field => ["fields_to_keep"]

    }
}
```

---

<div class="post-metadata">

**Author:** ![joserose](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@joserose](https://discuss.elastic.co/u/joserose)\
**Post date:** [April 5, 2016, 1:35pm UTC](https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063/4 "2016-04-05T13:35:28Z")

</div>

Thanks very much for the assistance thus far!

How do I specify which fields to keep in my

```
    match => ["message", "(?<fields_to_keep>{%{GREEDYDATA}})"]

```

line? Until my colleague makes a modification to the logged output I can get the entries that I'm after by performing the following:

`awk '{ print $11 $12 $13,$14 }' production.log > clean-production.log`

though obviously I want to perform that ongoing at the logstash ingress stage.

I've used the grok constructor but I'm not quite sure I understand it correctly; I don't want to establish the fields I don't want as any particular entry (e.g. I don't care about  
[2016-01-27T11:49:29.702997 #90] being [TOMCATDATESTAMP] etc), I just want to discard everything aside from the content between the {} inclusive...

Thanks again for the help!

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [April 5, 2016, 1:59pm UTC](https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063/5 "2016-04-05T13:59:49Z")

</div>

> How do I specify which fields to keep in my ... line?

That expression just extracts everything between "{" and "}" (including the braces), which should work as long as the first "{" occurrence is the start of the JSON string you're interested in.

> I've used the grok constructor but I'm not quite sure I understand it correctly; I don't want to establish the fields I don't want as any particular entry (e.g. I don't care about  
> [2016-01-27T11:49:29.702997 #90] being [TOMCATDATESTAMP] etc), I just want to discard everything aside from the content between the {} inclusive...

That's what @mick66's expression does. But again, if there are other curly braces in the string you might be in for a surprise.

---

<div class="post-metadata">

**Author:** ![joserose](https://avatars.discourse-cdn.com/v4/letter/j/da6949/32.png) [@joserose](https://discuss.elastic.co/u/joserose)\
**Post date:** [April 5, 2016, 4:38pm UTC](https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063/6 "2016-04-05T16:38:40Z")

</div>

Hi guys,

Thanks again for your help. @mick66 that's exactly what I was after, and @magnusbaeck thanks, I will make sure with my colleagues that there will be no unexpected braces anywhere!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:03am UTC](https://discuss.elastic.co/t/dropping-mutating-first-part-of-log-event/46063/7 "2017-07-06T05:03:49Z")

</div>