# Trim log line before JSON codec in Input?

**URL:** <https://discuss.elastic.co/t/trim-log-line-before-json-codec-in-input/176532>\
**Category:** Logstash\
**Created:** [April 12, 2019, 4:30am UTC](https://discuss.elastic.co/t/trim-log-line-before-json-codec-in-input/176532 "2019-04-12T04:30:03Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![danielmcquillen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmcquillen/32/23536_2.png) [@danielmcquillen](https://discuss.elastic.co/u/danielmcquillen)\
**Post date:** [April 12, 2019, 4:30am UTC](https://discuss.elastic.co/t/trim-log-line-before-json-codec-in-input/176532/1 "2019-04-12T04:30:04Z")

</div>

I was using the `cloudwatch_logs` plugin to read JSON-based logs from an AWS cloudwatch logstream. Those log lines were read ok, with the keys in the JSON arriving as properties on my event object and accessible in the filter section. (The log is from an Open edX installation on my server, not generated by CloudWatch.)

However, I've had to switch to using S3 to read these logs. When you save CloudWatch logs to S3, they get stored as a string with the timestamp and then the actual JSON string. Like

`2018-01-01T12:21:11.316Z {"username": "blah", .... }`

Is there a way to trim off the timestamp in my input section so I can still use a JSON codec? (Don't need that initial timestamp, there's one in the actual JSON log.)

Or do I have to transform the message into JSON in the filter section (using dissect?)? If so, I'm not sure how to do that so the data in the log JSON is set back to the top event's properties so I can still do something like

`if[username]=="blah"`

I thought I could do something like:

```auto
ruby {
        code => "event['message'] = event['message'][25..-1]"
    }

    json {
        source => "message"
        target => "message"
    }

```

but that doesn't work. (If it did, I'm assuming Logstash would have some internal way of automatically mapping the keys in message to the fields that the event object exposes, such that I could do the `[username]` conditional...but maybe I have that wrong too.)

Thanks for any suggestions.

---

<div class="post-metadata">

**Author:** ![A\_B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/a_b/32/17104_2.png) [@A\_B](https://discuss.elastic.co/u/A_B)\
**Post date:** [April 12, 2019, 7:15am UTC](https://discuss.elastic.co/t/trim-log-line-before-json-codec-in-input/176532/2 "2019-04-12T07:15:59Z")

</div>

Hi @danielmcquillen,

Something like this should work

```
  if [my_key] == "something" {
    grok {
        match => { "message" => "%{TIMESTAMP_ISO8601:timestamp}%{SPACE}%{GREEDYDATA:json_data}" }
    }
      json {
        source => "json_data"
    }
  } 

```

You can't really do `if [username]=="blah"` that way before you have parsed the JSON.

---

<div class="post-metadata">

**Author:** ![danielmcquillen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/danielmcquillen/32/23536_2.png) [@danielmcquillen](https://discuss.elastic.co/u/danielmcquillen)\
**Post date:** [April 12, 2019, 7:53pm UTC](https://discuss.elastic.co/t/trim-log-line-before-json-codec-in-input/176532/3 "2019-04-12T19:53:59Z")

</div>

Awesome. Thanks A\_B!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 10, 2019, 7:54pm UTC](https://discuss.elastic.co/t/trim-log-line-before-json-codec-in-input/176532/4 "2019-05-10T19:54:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.