# Parsing nested json logs iin logstash

**URL:** <https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718>\
**Category:** Logstash\
**Created:** [June 8, 2017, 1:38pm UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718 "2017-06-08T13:38:42Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [June 8, 2017, 1:38pm UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718/1 "2017-06-08T13:38:43Z")

</div>

I have a json which I need to store in elasticsearch,

Suppose this is the json,

```
{"name":"luvpreet","tasks":{"task1":"logstash","task2":"elastic"},"time":{"time1":"1week","time2":"2week"}}

```

And I want fields,

```
name : luvpreet
task1 : logstash
task2 : elastic
time1 : 1week
time2 : 2week

```

Will this filter below be enough,

```
filter{
json {
source => "message"
}
}

```

?

Please tell me how to do it ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 9, 2017, 8:51am UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718/2 "2017-06-09T08:51:41Z")

</div>

Is the number of task and time values fixed or will it vary arbitrarily? That is, will there always be `task1` and `task2` and never anything else?

---

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [June 9, 2017, 9:03am UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718/3 "2017-06-09T09:03:40Z")

</div>

Yes they will change.

Actually this is an example I made for showing purpose. My actual JSON is something else but in the same format.  
Yes, the tasks will change.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 9, 2017, 9:31am UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718/4 "2017-06-09T09:31:09Z")

</div>

The built-in filters for move fields don't have any loop or wildcard support so you'll have to write a small snippet of Ruby code in a ruby filter to move all subfields onto the top level. This isn't the first time the question is asked so you should be able to find further details in the archives.

---

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [June 9, 2017, 12:13pm UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718/5 "2017-06-09T12:13:08Z")

</div>

Okay, I have dropped that idea and now will use a different approach.

Here it is, the logs will be coming from rsyslog in the format below,

```
 {
   "@timestamp" : "",
    "message" : "34.201.66.248 - - [09/Jun/2017:17:02:19 +0530] "GET /api/v1/services/homepage/?site_id=8 HTTP/1.1" 200 1488 "-" "PostmanRuntime/6.1.6",
    "host" : "alpha",
    "programname" : "nginx",
    "procid" : "23421"
 }

```

Can I make different fields of the message ? Like IP, request, response, client ?

---

<div class="post-metadata">

**Author:** ![sLuvpreet33](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sluvpreet33/32/50626_2.png) [@sLuvpreet33](https://discuss.elastic.co/u/sLuvpreet33)\
**Post date:** [June 9, 2017, 12:14pm UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718/6 "2017-06-09T12:14:14Z")

</div>

> [@magnusbaeck](#):
>
> Is the number of task and time values fixed or will it vary arbitrarily? That is, will there always be task1 and task2 and never anything else?

And what if the task1 and task2 remain same ? Is it possible then ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [June 9, 2017, 12:28pm UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718/7 "2017-06-09T12:28:17Z")

</div>

> Can I make different fields of the message ? Like IP, request, response, client ?

Sure, parsing HTTP logs is a standard task. There are examples in the Logstash documentation.

> And what if the task1 and task2 remain same ? Is it possible then ?

Use the mutate filter's `rename` option to move fields.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 7, 2017, 12:28pm UTC](https://discuss.elastic.co/t/parsing-nested-json-logs-iin-logstash/88718/8 "2017-07-07T12:28:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.