# How to assign the extracted data from log file to new added fields

**URL:** <https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866>\
**Category:** Logstash\
**Created:** [August 30, 2020, 6:17am UTC](https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866 "2020-08-30T06:17:12Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![sudhakar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sudhakar/32/74789_2.png) [@sudhakar](https://discuss.elastic.co/u/sudhakar)\
**Post date:** [August 30, 2020, 6:17am UTC](https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866/1 "2020-08-30T06:17:12Z")

</div>

Hi ,

I have the following logstash config for adding 2 new fields "timestamp1" and "response1" , when I use mutate block I am able to see the new fields but the data parsing is failing from the log file.

> filter {  
> grok {  
> patterns\_dir =\> ["D:\hp\_deloitte\sudhsrivastava\ELK\_setup\logstash-6.0.1.OLD\patterns"]  
> match =\> { "message" =\> "%{GREEDYDATA:message}%{RESPPATTERN:response1}%{DATEPATTERN:timestamp1}"}  
> }  
> mutate {  
> add\_field =\> { "timestamp1" =\> "%{timestamp1}" }  
> add\_field =\> { "response1" =\> "%{response1}" }  
> }  
> date {  
> match =\> ["timestamp1", "yyyy-MM-dd HH:mm:ss,SSS"]  
> target =\> "timestamp1"  
> }

Below is the log sample:

```
`2020-08-02 10:53:02,052 DEB [0.TriggerInputAdapter:1)]] c.r.f.aAnalyticsApiRequestor -Sending API request :: ResponseTime : 6 ms - Recieved`

```

But in Kibana this is what i am seeing:

```
`timestamp1:%{timestamp1}` 
`response1:%{response1}`

```

However I am also getting the tags in kibana : `tags:beats_input_codec_plain_applied, _grokparsefailure, _dateparsefailure`

and not the actual values. Please help. Where am I gong wrong?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 30, 2020, 3:05pm UTC](https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866/2 "2020-08-30T15:05:39Z")

</div>

> [@sudhakar](#):
>
> mutate {  
> add\_field =\> { "timestamp1" =\> "%{timestamp1}" }  
> add\_field =\> { "response1" =\> "%{response1}" }  
> }

That sets the [timestamp1] field to the value of the [timestamp1] field (i.e. it does nothing) and similarly for [response1]. What are you trying to do with that?

If you are getting a \_grokparsefailure tag then your grok patterns do not match. How are RESPPATTERN and DATEPATTERN defined?

---

<div class="post-metadata">

**Author:** ![sudhakar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sudhakar/32/74789_2.png) [@sudhakar](https://discuss.elastic.co/u/sudhakar)\
**Post date:** [August 30, 2020, 3:29pm UTC](https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866/3 "2020-08-30T15:29:17Z")

</div>

Thanks @Badger for the response.

below are two patterns:

> RESPPATTERN [0-9]{1,2}ms =\> to match "6ms"  
> DATEPATTERN [0-9]{4}-[0-9]{2}-[0-9]{1,2}.?[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2},[0-9]{3} =\> to match timestamp

I am trying to match the timestamp and "6ms" in the log.

By using mutate i get the new fields in kibana but due to parse error values are not coming up.

how do I make logstash to ship/assign the extracted values to the new fields ?. As per what i understood mutate creates a new field and assign the value to it and grok patterns exatracts the matched value from logs.

Please suggest what changes do i need to make?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 30, 2020, 3:53pm UTC](https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866/4 "2020-08-30T15:53:05Z")

</div>

The grok pattern has to match the log message. Your message starts with DATEPATTERN, it does not end with it. Also, the '6 ms' has a space in it, so you could try

```
    grok {
        pattern_definitions => {
            "RESPPATTERN" => "[0-9]{1,2} ms"
            "DATEPATTERN" => "[0-9]{4}-[0-9]{2}-[0-9]{1,2}.?[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2},[0-9]{3}"
        }
        match => { "message" => "%{DATEPATTERN:timestamp1}%{GREEDYDATA:message}%{RESPPATTERN:response1}" }
    }
```

---

<div class="post-metadata">

**Author:** ![sudhakar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sudhakar/32/74789_2.png) [@sudhakar](https://discuss.elastic.co/u/sudhakar)\
**Post date:** [August 30, 2020, 4:28pm UTC](https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866/5 "2020-08-30T16:28:51Z")

</div>

Thanks @Badger.

I made the changes as per your suggestion: Below is the final logstash config:

> input {  
> beats {  
> port =\> 5044  
> type =\> "log"  
> host =\> "127.0.0.1"  
> }  
> }  
> output {  
> elasticsearch {  
> hosts =\> "127.0.0.1:9200"  
> manage\_template =\> false  
> index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
> document\_type =\> "%{[@metadata][type]}"  
> }  
> }  
> filter {  
> grok {  
> pattern\_definitions =\> {  
> "RESPPATTERN" =\> "[0-9]{1,2} ms"  
> "DATEPATTERN" =\> "[0-9]{4}-[0-9]{2}-[0-9]{1,2}.?[0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2},[0-9]{3}"  
> }  
> match =\> { "message" =\> "%{DATEPATTERN:timestamp1}%{GREEDYDATA:message}%{RESPPATTERN:response1}"}  
> }  
> mutate {  
> add\_field =\> { "timestamp2" =\> "%{timestamp1}" }  
> add\_field =\> { "response2" =\> "%{response1}" }  
> }  
> date {  
> match =\> ["timestamp2", "yyyy-MM-dd HH:mm:ss,SSS"]  
> target =\> "timestamp2"  
> }  
> }

I changed new field in mutate block. But I am still getting the date gropparse error and date parse errors.

---

<div class="post-metadata">

**Author:** ![sudhakar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sudhakar/32/74789_2.png) [@sudhakar](https://discuss.elastic.co/u/sudhakar)\
**Post date:** [August 30, 2020, 4:47pm UTC](https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866/6 "2020-08-30T16:47:17Z")

</div>

Ok, got it @Badger. Thanks.

I had to match the last part of the log message post 6ms also.

It is working now.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 27, 2020, 4:47pm UTC](https://discuss.elastic.co/t/how-to-assign-the-extracted-data-from-log-file-to-new-added-fields/246866/7 "2020-09-27T16:47:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.