# Manipulate data with grok - logstash

**URL:** <https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755>\
**Category:** Logstash\
**Created:** [January 25, 2017, 10:24am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755 "2017-01-25T10:24:23Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 25, 2017, 10:24am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/1 "2017-01-25T10:24:23Z")

</div>

hi, i have log data like this.

> 2017-01-19T18:08:35+07:00 payment INFO {"user\_id":0,"cart\_id":"81746"}

so i want using logstash to input data to elasticsearch.  
but i want to manipulate the data it self before insert it to elastic.

so i use this tool to help me  
[http://grokdebug.herokuapp.com/](http://grokdebug.herokuapp.com/)

here is my pattern.

> %{TIMESTAMP\_ISO8601} %{GREEDYDATA:message} %{LOGLEVEL:log-level} %{GREEDYDATA:json}

and here is the data that i got after debug this log

> {  
> "TIMESTAMP\_ISO8601": [  
> [  
> "2017-01-19T18:08:35+07:00"  
> ]  
> ],  
> "YEAR": [  
> [  
> "2017"  
> ]  
> ],  
> "MONTHNUM": [  
> [  
> "01"  
> ]  
> ],  
> "MONTHDAY": [  
> [  
> "19"  
> ]  
> ],  
> "HOUR": [  
> [  
> "18",  
> "07"  
> ]  
> ],  
> "MINUTE": [  
> [  
> "08",  
> "00"  
> ]  
> ],  
> "SECOND": [  
> [  
> "35"  
> ]  
> ],  
> "ISO8601\_TIMEZONE": [  
> [  
> "+07:00"  
> ]  
> ],  
> "message": [  
> [  
> "payment"  
> ]  
> ],  
> "log": [  
> [  
> "INFO"  
> ]  
> ],  
> "json": [  
> [  
> "{"user\_id":0,"cart\_id":"81746"}"  
> ]  
> ]  
> }

so what i want next is manipulate data,  
lets say, i want to take :message, :log-leve, and :json  
and combine it, and then make one json object with that values.

how can i do that?  
pls help me.

thank you

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 25, 2017, 11:06am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/2 "2017-01-25T11:06:04Z")

</div>

Given the example log line

```
2017-01-19T18:08:35+07:00 payment INFO {"user_id":0,"cart_id":"81746"}

```

what do you want the resulting event to look like? If you want a specific answer you need to be specific about what you want to do.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 25, 2017, 9:12pm UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/3 "2017-01-25T21:12:17Z")

</div>

i want something like this :  
{  
'logtime': '2017-01-19T18:08:35+07:00',  
'message':'payment',  
'log-level':'INFO',  
'json': {  
'user\_id':'0',  
'cart\_id':'81746'  
}  
}

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 26, 2017, 6:49am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/4 "2017-01-26T06:49:23Z")

</div>

Use what you already have but add a json filter that parses the `json` field and stores the result back into the `json` field.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 26, 2017, 7:49am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/5 "2017-01-26T07:49:28Z")

</div>

hi,

i did this in my logstash conf

> filter {  
> grok {  
> match =\> { "message" =\> "%{TIMESTAMP\_ISO8601} %{GREEDYDATA:message} %{LOGLEVEL:log-level} %{GREEDYDATA:json}" }  
> }

> ```
> json { source => message }
> 
> ```
> 
> }

the data successfully added to elastic.  
but the format, its not what i want.

i got this :

> ```
> "path": "/usr/local/Cellar/logstash/5.1.1/payment.json",
> "@timestamp": "2017-01-26T07:45:16.986Z",
> "log-level": "INFO",
> "@version": "1",
> "host": "Gutas-MBP",
> "json": " {\"user_id\":0,\"cart_id\":\"81746\"}",
> "message": [
> "2017-01-19T18:08:35+07:00 payment INFO {\"user_id\":0,\"cart_id\":\"81746\"}",
> "payment"
> ]
> 
> ```

what i want is something like this :

```
  "path": "/usr/local/Cellar/logstash/5.1.1/payment.json",
  "@timestamp": "2017-01-26T07:45:16.986Z",
  "log-level": "INFO",
  "@version": "1",
  "host": "Gutas-MBP",
  "json": [
        "user_id ": 0,
        "cart_id":17676,
          "TIMESTAMP_ISO8601": "2017-01-19T18:08:35+07:00",
         "message": "payment",
        "log_level": "INFO"
  ]

```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 26, 2017, 8:20am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/6 "2017-01-26T08:20:15Z")

</div>

Use the json filter's `target` option to control where the parsed JSON values are stored.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 26, 2017, 8:29am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/7 "2017-01-26T08:29:50Z")

</div>

i tried this :

> filter {  
> grok {  
> match =\> { "message" =\> "%{TIMESTAMP\_ISO8601} %{GREEDYDATA:message} %{LOGLEVEL:log-level} %{GREEDYDATA:json}" }  
> }

> json {  
> source =\> "message"  
> target =\> "message"  
> }

> }

i got something like this :

> ```
> "message": [
> "2017-01-19T18:08:35+07:00 payment INFO {\"user_id\":0,\"cart_id\":\"81746\"}",
> "payment"
> ],
> 
> ```

i want something like this

message : [  
'logtime' : '2017-01-19T18:08:35+07:00'  
'message': 'payment'  
'log\_level': 'INFO',  
'json': [  
'user\_id': '3313',  
'cart\_id': '222'  
]  
]

what should i do sir?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 26, 2017, 9:27am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/8 "2017-01-26T09:27:10Z")

</div>

Why are you parsing the `message` field when your JSON data is in the `json` field?

Secondly, you need `overwrite => ["message"]` in your grok filter so that it's allowed to overwrite the current contents of the `message` field.

---

<div class="post-metadata">

**Author:** ![gutasaputra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gutasaputra/32/10472_2.png) [@gutasaputra](https://discuss.elastic.co/u/gutasaputra)\
**Post date:** [January 26, 2017, 10:02am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/9 "2017-01-26T10:02:09Z")

</div>

no, i want to combine these 4 objects, into one big json object.  
i want this log

> 2017-01-19T18:08:35+07:00 payment INFO {"user\_id":0,"cart\_id":"81746"}

and convert it all to one json object.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 26, 2017, 10:55am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/10 "2017-01-26T10:55:50Z")

</div>

Oh. Well, you can use a mutate filter to rename fields, including moving fields to become subfields. I also believe you can reference nested fields in the grok filter to put them in the right place from the start.

```
%{TIMESTAMP_ISO8601} %{GREEDYDATA:[message][message]} %{LOGLEVEL:[message][log-level]} %{GREEDYDATA:json}

```

The `[field][subfield]` notation can be used in the json filter too. See [https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) for more on field references.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 23, 2017, 10:55am UTC](https://discuss.elastic.co/t/manipulate-data-with-grok-logstash/72755/11 "2017-02-23T10:55:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.