How to parse nested logs with JSON through logstash

Elastic Stack Logstash
1

Hello All,

I am facing issue while parsing nested JSON in my logs. I checked some of the posts over internet regarding nested JSON but none actually resolved my problem. There might be the case I am making some mistake and need help from the community experts.

I am sending JSON logs using Filebeat -> LogStash -> ElasticSearch. Logstash doesn't parse the nested JSON correctly and it sends data as

application_log : %{[parsedJson][application_log]}

Below is my JSON

"_type": "syslog",  "_source": {    "@version": "1",    "@timestamp": "2020-07-17T02:31:22.884Z",    "type": "syslog",    "host": "172.29.240.11",    "syslog_pri": "30",    "syslog_timestamp": "Jul 17 03:31:22",    "docker_image": "test:5000/app-test-caas:1.0.128-d7a033f8",    "docker_container": "STG-test-caas-app",    "docker_container_id": "e553d70469cf",    "syslog_pid": "4875",    "received_at": [      "2020-07-17T02:31:22.884Z",      "2020-07-17T02:31:22.884Z"    ],    "received_from": [      "172.29.240.11",      "172.29.240.11"    ],    "application_log": {      "hostname": "e553d70469cf",      "timestamp": "2020-07-17T02:31:22.884Z",      "service": "wealth-testuk-caas",      "category": "unknown",      "level": "error",      "meta": {        "stack": [          "StatusCodeError: 500 - \"The website encountered an unexpected error. Please try again later.<br />\"",          "    at new StatusCodeError (/data/app/node_modules/request-promise-core/lib/errors.js:32:15)",          "    at Request.plumbing.callback (/data/app/node_modules/request-promise-core/lib/plumbing.js:104:33)",          "    at Request.RP$callback [as _callback] (/data/app/node_modules/request-promise-core/lib/plumbing.js:46:31)",          "    at Request.self.callback (/data/app/node_modules/request/request.js:185:22)",          "    at emitTwo (events.js:106:13)",          "    at Request.emit (events.js:191:7)",          "    at Request.<anonymous> (/data/app/node_modules/request/request.js:1154:10)",          "    at emitOne (events.js:96:13)",          "    at Request.emit (events.js:188:7)",          "    at IncomingMessage.<anonymous> (/data/app/node_modules/request/request.js:1076:12)"        ],        "method": "GET",        "url": "/resources/outage"      },      "message": "500 - \"The website encountered an unexpected error. Please try again later.<br />\""    },    "tags": [      "app-json"    ]  },  "fields": {    "@timestamp": [      1594953082884    ],    "application_log.timestamp": [      1594953082884    ],    "received_at": [      1594953082884,      1594953082884    ]  },  "highlight": {    "docker_container": [      "STG-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas-app"    ],    "docker_image": [      "test:5000/app-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas:1.0.128-d7a033f8"    ]  },  "sort": [    1594953082884  ]

}

Below is my logstash configuration:

input {
beats {
port => 5044
}
}
filter {
json
{
source => "message"
target => "parsedJson"
}
mutate
{
add_field =>
{
"syslog_message" => "%{[parsedJson][syslog_message]}"
"application_log" => "%{[parsedJson][application_log]}"
}
}

}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
codec => "json"

}
}

Expected output:

application_log.category: "unknown"
application_log.level : "error"
application_log.message:"500 - The website encountered an unexpected error. Please try again later."

Please guide me with the logstash configurations and/or other options to parse the logs to achieve expected output.

Do let me know in case any information is required to help me on this case.

Thanks,
Ranjan gupta

2

Neither application_log nor syslog_message exists in your JSON, so this is expected.

3

Hi Badger,

Application_log field is there in the JSON, you may scroll towards right.

I attached below the JSON logs again for quick reference:

`"
{
"_type": "syslog",
"_source":
{
"@version": "1"
, "@timestamp": "2020-07-17T02:31:22.884Z"
, "type": "syslog"
, "host": "172.29.240.11"
, "syslog_pri": "30"
, "syslog_timestamp": "Jul 17 03:31:22"
, "docker_image": "test:5000/app-test-caas:1.0.128-d7a033f8"
, "docker_container": "STG-test-caas-app"
, "docker_container_id": "e553d70469cf"
, "syslog_pid": "4875"
, "received_at": [ "2020-07-17T02:31:22.884Z", "2020-07-17T02:31:22.884Z" ]
, "received_from": [ "172.29.240.11", "172.29.240.11" ]
, "application_log":
{
"hostname": "e553d70469cf"
, "timestamp": "2020-07-17T02:31:22.884Z"
, "service": "wealth-testuk-caas"
, "category": "unknown"
, "level": "error"
, "meta": {
"stack": ["StatusCodeError: 500 - "The website encountered an unexpected error. Please try again later.
"", " at new StatusCodeError (/data/app/node_modules/request-promise-core/lib/errors.js:32:15)", " at Request.plumbing.callback (/data/app/node_modules/request-promise-core/lib/plumbing.js:104:33)", " at Request.RP$callback [as _callback] (/data/app/node_modules/request-promise-core/lib/plumbing.js:46:31)", " at Request.self.callback (/data/app/node_modules/request/request.js:185:22)", " at emitTwo (events.js:106:13)", " at Request.emit (events.js:191:7)", " at Request. (/data/app/node_modules/request/request.js:1154:10)", " at emitOne (events.js:96:13)", " at Request.emit (events.js:188:7)", " at IncomingMessage. (/data/app/node_modules/request/request.js:1076:12)" ]
, "method": "GET"
, "url": "/resources/outage"
}
, "message": "500 - "The website encountered an unexpected error. Please try again later.
""
}
, "tags": [ "app-json" ]
}
, "fields": { "@timestamp": [ 1594953082884 ], "application_log.timestamp": [ 1594953082884 ], "received_at": [ 1594953082884, 1594953082884 ] }
, "highlight": { "docker_container": [ "STG-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas-app" ], "docker_image": [ "test:5000/app-@kibana-highlighted-field@test@/kibana-highlighted-field@-caas:1.0.128-d7a033f8" ] }
, "sort": [ 1594953082884 ]

}
"`

4

If I run

filter { json { source => "message" target => "parsedJson" remove_field => [ "message" ] } }
input { generator { count => 1 lines => [ '{ "type": "syslog", "syslog_pri": "30",    "syslog_timestamp": "Jul 17 03:31:22",    "docker_image": "test:5000/app-test-caas:1.0.128-d7a033f8",    "docker_container": "STG-test-caas-app",    "docker_container_id": "e553d70469cf",    "syslog_pid": "4875",    "received_at": [      "2020-07-17T02:31:22.884Z",      "2020-07-17T02:31:22.884Z"    ],    "received_from": [      "172.29.240.11",      "172.29.240.11"    ],    "application_log": {      "hostname": "e553d70469cf",      "timestamp": "2020-07-17T02:31:22.884Z",      "service": "wealth-testuk-caas",      "category": "unknown",      "level": "error",      "meta": {        "stack": [          "StatusCodeError: 500 - \"The website encountered an unexpected error. Please try again later.<br />\"",          "    at new StatusCodeError (/data/app/node_modules/request-promise-core/lib/errors.js:32:15)",          "    at Request.plumbing.callback (/data/app/node_modules/request-promise-core/lib/plumbing.js:104:33)",          "    at Request.RP$callback [as _callback] (/data/app/node_modules/request-promise-core/lib/plumbing.js:46:31)",          "    at Request.self.callback (/data/app/node_modules/request/request.js:185:22)",          "    at emitTwo (events.js:106:13)",          "    at Request.emit (events.js:191:7)",          "    at Request.<anonymous> (/data/app/node_modules/request/request.js:1154:10)",          "    at emitOne (events.js:96:13)",          "    at Request.emit (events.js:188:7)",          "    at IncomingMessage.<anonymous> (/data/app/node_modules/request/request.js:1076:12)"        ],        "method": "GET",        "url": "/resources/outage"      },      "message": "500 - \"The website encountered an unexpected error. Please try again later.<br />\""    }}' ] } }
filter {
    mutate {
        add_field => {
            "syslog_message" => "%{[parsedJson][syslog_message]}"
            "application_log" => "%{[parsedJson][application_log]}"
        }
    }
}
output  { stdout { codec => rubydebug { metadata => false } } }

then I get

"application_log" => "{\"hostname\":\"e553d70469cf\",\"level\":\"error\",\"meta\":{\"method\":\"GET\",\"url\":\"/resources/outage\",\"stack\":[\"StatusCodeError: 500 - \\\"The website encountered an unexpected error. Please try again later.<br />\\\"\",\"    at new StatusCodeError (/data/app/node_modules/request-promise-core/lib/errors.js:32:15)\",\"    at Request.plumbing.callback (/data/app/node_modules/request-promise-core/lib/plumbing.js:104:33)\",\"    at Request.RP$callback [as _callback] (/data/app/node_modules/request-promise-core/lib/plumbing.js:46:31)\",\"    at Request.self.callback (/data/app/node_modules/request/request.js:185:22)\",\"    at emitTwo (events.js:106:13)\",\"    at Request.emit (events.js:191:7)\",\"    at Request.<anonymous> (/data/app/node_modules/request/request.js:1154:10)\",\"    at emitOne (events.js:96:13)\",\"    at Request.emit (events.js:188:7)\",\"    at IncomingMessage.<anonymous> (/data/app/node_modules/request/request.js:1076:12)\"]},\"service\":\"wealth-testuk-caas\",\"category\":\"unknown\",\"message\":\"500 - \\\"The website encountered an unexpected error. Please try again later.<br />\\\"\",\"timestamp\":\"2020-07-17T02:31:22.884Z\"}",
 "syslog_message" => "%{[parsedJson][syslog_message]}",

so I think your description of what you are doing is incorrect.

5

Thanks Badger, I got to know where I was making mistake. I need to pass the whole JSON log in single line for logstash to parse them easily. Thanks for your help.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.