# How to parse nested logs with JSON through logstash

**URL:** <https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781>\
**Category:** Logstash\
**Created:** [July 19, 2020, 10:40am UTC](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781 "2020-07-19T10:40:57Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Ranjan\_Bansal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjan_bansal/32/48333_2.png) [@Ranjan\_Bansal](https://discuss.elastic.co/u/Ranjan_Bansal)\
**Post date:** [July 19, 2020, 10:40am UTC](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781/1 "2020-07-19T10:40:57Z")

</div>

Hello All,

I am facing issue while parsing nested JSON in my logs. I checked some of the posts over internet regarding nested JSON but none actually resolved my problem. There might be the case I am making some mistake and need help from the community experts.

I am sending JSON logs using Filebeat -\> LogStash -\> ElasticSearch. Logstash doesn't parse the nested JSON correctly and it sends data as

application\_log : %{[parsedJson][application\_log]}

Below is my JSON

* * *

```
"_type": "syslog", "_source": { "@version": "1", "@timestamp": "2020-07-17T02:31:22.884Z", "type": "syslog", "host": "172.29.240.11", "syslog_pri": "30", "syslog_timestamp": "Jul 17 03:31:22", "docker_image": "test:5000/app-test-caas:1.0.128-d7a033f8", "docker_container": "STG-test-caas-app", "docker_container_id": "e553d70469cf", "syslog_pid": "4875", "received_at": ["2020-07-17T02:31:22.884Z", "2020-07-17T02:31:22.884Z"], "received_from": ["172.29.240.11", "172.29.240.11"], "application_log": { "hostname": "e553d70469cf", "timestamp": "2020-07-17T02:31:22.884Z", "service": "wealth-testuk-caas", "category": "unknown", "level": "error", "meta": { "stack": ["StatusCodeError: 500 - \"The website encountered an unexpected error. Please try again later.<br />\"", " at new StatusCodeError (/data/app/node_modules/request-promise-core/lib/errors.js:32:15)", " at Request.plumbing.callback (/data/app/node_modules/request-promise-core/lib/plumbing.js:104:33)", " at Request.RP$callback [as _callback] (/data/app/node_modules/request-promise-core/lib/plumbing.js:46:31)", " at Request.self.callback (/data/app/node_modules/request/request.js:185:22)", " at emitTwo (events.js:106:13)", " at Request.emit (events.js:191:7)", " at Request.<anonymous> (/data/app/node_modules/request/request.js:1154:10)", " at emitOne (events.js:96:13)", " at Request.emit (events.js:188:7)", " at IncomingMessage.<anonymous> (/data/app/node_modules/request/request.js:1076:12)" ], "method": "GET", "url": "/resources/outage" }, "message": "500 - \"The website encountered an unexpected error. Please try again later.<br />\"" }, "tags": ["app-json"] }, "fields": { "@timestamp": [1594953082884], "application_log.timestamp": [1594953082884], "received_at": [1594953082884, 1594953082884] }, "highlight": { "docker_container": ["STG-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas-app"], "docker_image": ["test:5000/app-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas:1.0.128-d7a033f8"] }, "sort": [1594953082884]

```

}

* * *

Below is my logstash configuration:

* * *

input {  
beats {  
port =\> 5044  
}  
}  
filter {  
json  
{  
source =\> "message"  
target =\> "parsedJson"  
}  
mutate  
{  
add\_field =\>  
{  
"syslog\_message" =\> "%{[parsedJson][syslog\_message]}"  
"application\_log" =\> "%{[parsedJson][application\_log]}"  
}  
}

}  
output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"  
codec =\> "json"

## } }

Expected output:

application\_log.category: "unknown"  
application\_log.level : "error"  
application\_log.message:"500 - The website encountered an unexpected error. Please try again later."

Please guide me with the logstash configurations and/or other options to parse the logs to achieve expected output.

Do let me know in case any information is required to help me on this case.

Thanks,  
Ranjan gupta

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 19, 2020, 6:12pm UTC](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781/2 "2020-07-19T18:12:42Z")

</div>

> [@Ranjan\_Bansal](#):
>
> application\_log : %{[parsedJson][application\_log]}

Neither application\_log nor syslog\_message exists in your JSON, so this is expected.

---

<div class="post-metadata">

**Author:** ![Ranjan\_Bansal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjan_bansal/32/48333_2.png) [@Ranjan\_Bansal](https://discuss.elastic.co/u/Ranjan_Bansal)\
**Post date:** [July 20, 2020, 3:06am UTC](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781/3 "2020-07-20T03:06:35Z")

</div>

Hi Badger,

Application\_log field is there in the JSON, you may scroll towards right.

I attached below the JSON logs again for quick reference:

`"  
{  
"\_type": "syslog",  
"\_source":  
{  
"@version": "1"  
, "@timestamp": "2020-07-17T02:31:22.884Z"  
, "type": "syslog"  
, "host": "172.29.240.11"  
, "syslog\_pri": "30"  
, "syslog\_timestamp": "Jul 17 03:31:22"  
, "docker\_image": "test:5000/app-test-caas:1.0.128-d7a033f8"  
, "docker\_container": "STG-test-caas-app"  
, "docker\_container\_id": "e553d70469cf"  
, "syslog\_pid": "4875"  
, "received\_at": ["2020-07-17T02:31:22.884Z", "2020-07-17T02:31:22.884Z"]  
, "received\_from": ["172.29.240.11", "172.29.240.11"]  
, "application\_log":  
{  
"hostname": "e553d70469cf"  
, "timestamp": "2020-07-17T02:31:22.884Z"  
, "service": "wealth-testuk-caas"  
, "category": "unknown"  
, "level": "error"  
, "meta": {  
"stack": ["StatusCodeError: 500 - "The website encountered an unexpected error. Please try again later.  
"", " at new StatusCodeError (/data/app/node\_modules/request-promise-core/lib/errors.js:32:15)", " at Request.plumbing.callback (/data/app/node\_modules/request-promise-core/lib/plumbing.js:104:33)", " at Request.RP$callback [as \_callback] (/data/app/node\_modules/request-promise-core/lib/plumbing.js:46:31)", " at Request.self.callback (/data/app/node\_modules/request/request.js:185:22)", " at emitTwo (events.js:106:13)", " at Request.emit (events.js:191:7)", " at Request. (/data/app/node\_modules/request/request.js:1154:10)", " at emitOne (events.js:96:13)", " at Request.emit (events.js:188:7)", " at IncomingMessage. (/data/app/node\_modules/request/request.js:1076:12)" ]  
, "method": "GET"  
, "url": "/resources/outage"  
}  
, "message": "500 - "The website encountered an unexpected error. Please try again later.  
""  
}  
, "tags": ["app-json"]  
}  
, "fields": { "@timestamp": [1594953082884], "application\_log.timestamp": [1594953082884], "received\_at": [1594953082884, 1594953082884] }  
, "highlight": { "docker\_container": ["STG-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas-app"], "docker\_image": ["test:5000/app-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas:1.0.128-d7a033f8"] }  
, "sort": [1594953082884]

}  
"`

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 20, 2020, 5:38pm UTC](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781/4 "2020-07-20T17:38:06Z")

</div>

If I run

```
filter { json { source => "message" target => "parsedJson" remove_field => ["message"] } }
input { generator { count => 1 lines => ['{ "type": "syslog", "syslog_pri": "30", "syslog_timestamp": "Jul 17 03:31:22", "docker_image": "test:5000/app-test-caas:1.0.128-d7a033f8", "docker_container": "STG-test-caas-app", "docker_container_id": "e553d70469cf", "syslog_pid": "4875", "received_at": [ "2020-07-17T02:31:22.884Z", "2020-07-17T02:31:22.884Z"], "received_from": ["172.29.240.11", "172.29.240.11"], "application_log": { "hostname": "e553d70469cf", "timestamp": "2020-07-17T02:31:22.884Z", "service": "wealth-testuk-caas", "category": "unknown", "level": "error", "meta": { "stack": ["StatusCodeError: 500 - \"The website encountered an unexpected error. Please try again later.<br />\"", " at new StatusCodeError (/data/app/node_modules/request-promise-core/lib/errors.js:32:15)", " at Request.plumbing.callback (/data/app/node_modules/request-promise-core/lib/plumbing.js:104:33)", " at Request.RP$callback [as _callback] (/data/app/node_modules/request-promise-core/lib/plumbing.js:46:31)", " at Request.self.callback (/data/app/node_modules/request/request.js:185:22)", " at emitTwo (events.js:106:13)", " at Request.emit (events.js:191:7)", " at Request.<anonymous> (/data/app/node_modules/request/request.js:1154:10)", " at emitOne (events.js:96:13)", " at Request.emit (events.js:188:7)", " at IncomingMessage.<anonymous> (/data/app/node_modules/request/request.js:1076:12)" ], "method": "GET", "url": "/resources/outage" }, "message": "500 - \"The website encountered an unexpected error. Please try again later.<br />\"" }}' ] } }
filter {
    mutate {
        add_field => {
            "syslog_message" => "%{[parsedJson][syslog_message]}"
            "application_log" => "%{[parsedJson][application_log]}"
        }
    }
}
output { stdout { codec => rubydebug { metadata => false } } }

```

then I get

```
"application_log" => "{\"hostname\":\"e553d70469cf\",\"level\":\"error\",\"meta\":{\"method\":\"GET\",\"url\":\"/resources/outage\",\"stack\":[\"StatusCodeError: 500 - \\\"The website encountered an unexpected error. Please try again later.<br />\\\"\",\" at new StatusCodeError (/data/app/node_modules/request-promise-core/lib/errors.js:32:15)\",\" at Request.plumbing.callback (/data/app/node_modules/request-promise-core/lib/plumbing.js:104:33)\",\" at Request.RP$callback [as _callback] (/data/app/node_modules/request-promise-core/lib/plumbing.js:46:31)\",\" at Request.self.callback (/data/app/node_modules/request/request.js:185:22)\",\" at emitTwo (events.js:106:13)\",\" at Request.emit (events.js:191:7)\",\" at Request.<anonymous> (/data/app/node_modules/request/request.js:1154:10)\",\" at emitOne (events.js:96:13)\",\" at Request.emit (events.js:188:7)\",\" at IncomingMessage.<anonymous> (/data/app/node_modules/request/request.js:1076:12)\"]},\"service\":\"wealth-testuk-caas\",\"category\":\"unknown\",\"message\":\"500 - \\\"The website encountered an unexpected error. Please try again later.<br />\\\"\",\"timestamp\":\"2020-07-17T02:31:22.884Z\"}",
 "syslog_message" => "%{[parsedJson][syslog_message]}",

```

so I think your description of what you are doing is incorrect.

---

<div class="post-metadata">

**Author:** ![Ranjan\_Bansal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjan_bansal/32/48333_2.png) [@Ranjan\_Bansal](https://discuss.elastic.co/u/Ranjan_Bansal)\
**Post date:** [July 22, 2020, 4:59am UTC](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781/5 "2020-07-22T04:59:04Z")

</div>

Thanks Badger, I got to know where I was making mistake. I need to pass the whole JSON log in single line for logstash to parse them easily. Thanks for your help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 19, 2020, 4:59am UTC](https://discuss.elastic.co/t/how-to-parse-nested-logs-with-json-through-logstash/241781/6 "2020-08-19T04:59:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.