Hello All,
I am facing issue while parsing nested JSON in my logs. I checked some of the posts over internet regarding nested JSON but none actually resolved my problem. There might be the case I am making some mistake and need help from the community experts.
I am sending JSON logs using Filebeat -> LogStash -> ElasticSearch. Logstash doesn't parse the nested JSON correctly and it sends data as
application_log : %{[parsedJson][application_log]}
Below is my JSON
"_type": "syslog", "_source": { "@version": "1", "@timestamp": "2020-07-17T02:31:22.884Z", "type": "syslog", "host": "172.29.240.11", "syslog_pri": "30", "syslog_timestamp": "Jul 17 03:31:22", "docker_image": "test:5000/app-test-caas:1.0.128-d7a033f8", "docker_container": "STG-test-caas-app", "docker_container_id": "e553d70469cf", "syslog_pid": "4875", "received_at": [ "2020-07-17T02:31:22.884Z", "2020-07-17T02:31:22.884Z" ], "received_from": [ "172.29.240.11", "172.29.240.11" ], "application_log": { "hostname": "e553d70469cf", "timestamp": "2020-07-17T02:31:22.884Z", "service": "wealth-testuk-caas", "category": "unknown", "level": "error", "meta": { "stack": [ "StatusCodeError: 500 - \"The website encountered an unexpected error. Please try again later.<br />\"", " at new StatusCodeError (/data/app/node_modules/request-promise-core/lib/errors.js:32:15)", " at Request.plumbing.callback (/data/app/node_modules/request-promise-core/lib/plumbing.js:104:33)", " at Request.RP$callback [as _callback] (/data/app/node_modules/request-promise-core/lib/plumbing.js:46:31)", " at Request.self.callback (/data/app/node_modules/request/request.js:185:22)", " at emitTwo (events.js:106:13)", " at Request.emit (events.js:191:7)", " at Request.<anonymous> (/data/app/node_modules/request/request.js:1154:10)", " at emitOne (events.js:96:13)", " at Request.emit (events.js:188:7)", " at IncomingMessage.<anonymous> (/data/app/node_modules/request/request.js:1076:12)" ], "method": "GET", "url": "/resources/outage" }, "message": "500 - \"The website encountered an unexpected error. Please try again later.<br />\"" }, "tags": [ "app-json" ] }, "fields": { "@timestamp": [ 1594953082884 ], "application_log.timestamp": [ 1594953082884 ], "received_at": [ 1594953082884, 1594953082884 ] }, "highlight": { "docker_container": [ "STG-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas-app" ], "docker_image": [ "test:5000/app-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas:1.0.128-d7a033f8" ] }, "sort": [ 1594953082884 ]
}
Below is my logstash configuration:
input {
beats {
port => 5044
}
}
filter {
json
{
source => "message"
target => "parsedJson"
}
mutate
{
add_field =>
{
"syslog_message" => "%{[parsedJson][syslog_message]}"
"application_log" => "%{[parsedJson][application_log]}"
}
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
codec => "json"
}
}
Expected output:
application_log.category: "unknown"
application_log.level : "error"
application_log.message:"500 - The website encountered an unexpected error. Please try again later."
Please guide me with the logstash configurations and/or other options to parse the logs to achieve expected output.
Do let me know in case any information is required to help me on this case.
Thanks,
Ranjan gupta