Unable to parse nested json using logstash json filter

Elastic Stack Logstash
1

Hello All,

I am trying to parse logs which are in json format. I am trying to parse the message json field in the logs. I have tried using json plugin filter but it is not working as expected. Here is the log -

   {
    "id": "34984508843661791320106695906064879945445712328418263040",
    "timestamp": 1568759632569,
    "message": "{\"source_host\":\"ip-10-120-120-234.ec2.internal\",\"method\":\"newConnection\",\"level\":\"INFO\",\"appName\":\"testing\",\"message\":\"Attempting to open connection #1 to MySql\",\"mdc\":{},\"@timestamp\":\"2019-09-17T22:33:52.567Z\",\"file\":\"CachedConnectionProvider.java\",\"appGroup\":\"quality\",\"line_number\":\"88\",\"thread_name\":\"pool-17-thread-17\",\"@version\":1,\"logger_name\":\"io.regex.connect.jdbc.util.CachedConnectionProvider\",\"class\":\"io.regex.connect.jdbc.util.CachedConnectionProvider\"}",
    "aws": {
        "awslogs": {
            "logGroup": "testing",
            "logStream": "testing/c78919c03baa4e1fa75adf06cf6d4139",
            "owner": "2313245145",
            "account": "dev"
        },
        "tags": "awslogs"
    }
}

This is my logstash configuration -

    filter {
      json {
        source => "message"
      }
    }

Any help would be much appreciated.

Thanks in Advance

2

If that is actually an example of what you are trying to parse then it does not have a [message] field, it has a [_source][message] field. Normally I would assume that you are showing an example of the document in elasticsearch that results, but in that case the filter you show would parse it, so it suggests that assumption is incorrect.

Please clarify your question.

3

Sorry! My bad, I have posted the wrong sample log. Please have a look now, I have updated the log to the correct one. I am looking to parse message field.

4

And even then that filter will parse that JSON. What exactly do you mean by "is not working as expected"? If you are getting an error message then please post that. If the parsing looks wrong then you need to be much more detailed.

You have to provide enough information for us to help you.

5

I was getting _jsonparsefailure error. In the output section, I have enabled rubydebug and gave the above log event as stdin. Here is the error.log below. ( Couldn't attach complete log due to body limit to 7000 characters)

    }[2019-09-17T19:42:44,649][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"        \"tags\": \"awslogs\"", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"        "tags": "awslogs""; line: 1, column: 16]>}
    [2019-09-17T19:42:44,649][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"id\": \"34984508843661791320106695906064879945445712328418263040\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"    "id": "34984508843661791320106695906064879945445712328418263040","; line: 1, column: 10]>}
    [2019-09-17T19:42:44,649][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"aws\": {", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"    "aws": {"; line: 1, column: 11]>}
    [2019-09-17T19:42:44,649][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"            \"logGroup\": \"testing\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"            "logGroup": "testing","; line: 1, column: 24]>}
    [2019-09-17T19:42:44,649][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"        },", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"        },"; line: 1, column: 0])
     at [Source: (byte[])"        },"; line: 1, column: 10]>}
    [2019-09-17T19:42:44,649][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"message\": \"{\\\"source_host\\\":\\\"ip-10-120-120-234.ec2.internal\\\",\\\"method\\\":\\\"newConnection\\\",\\\"level\\\":\\\"INFO\\\",\\\"appName\\\":\\\"testing\\\",\\\"message\\\":\\\"Attempting to open connection #1 to MySql\\\",\\\"mdc\\\":{},\\\"@timestamp\\\":\\\"2019-09-17T22:33:52.567Z\\\",\\\"file\\\":\\\"CachedConnectionProvider.java\\\",\\\"appGroup\\\":\\\"quality\\\",\\\"line_number\\\":\\\"88\\\",\\\"thread_name\\\":\\\"pool-17-thread-17\\\",\\\"@version\\\":1,\\\"logger_name\\\":\\\"io.regex.connect.jdbc.util.CachedConnectionProvider\\\",\\\"class\\\":\\\"io.regex.connect.jdbc.util.CachedConnectionProvider\\\"}\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"    "message": "{\"source_host\":\"ip-10-120-120-234.ec2.internal\",\"method\":\"newConnection\",\"level\":\"INFO\",\"appName\":\"testing\",\"message\":\"Attempting to open connection #1 to MySql\",\"mdc\":{},\"@timestamp\":\"2019-09-17T22:33:52.567Z\",\"file\":\"CachedConnectionProvider.java\",\"appGroup\":\"quality\",\"line_number\":\"88\",\"thread_name\":\"pool-17-thread-17\",\"@version\":1,\"logger_name\":\"io.regex.connect.jdbc.util.CachedConnectionProvider\",\"class\":\"io.regex.connect.jd"[truncated 37 bytes]; line: 1, column: 15]>}
    [2019-09-17T19:42:44,650][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"            \"owner\": \"2313245145\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"            "owner": "2313245145","; line: 1, column: 21]>}
    [2019-09-17T19:42:44,650][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"        \"awslogs\": {", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"        "awslogs": {"; line: 1, column: 19]>}
    [2019-09-17T19:42:44,650][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    \"timestamp\": 1568759632569,", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"    "timestamp": 1568759632569,"; line: 1, column: 17]>}
    [2019-09-17T19:42:44,650][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"   {", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"   {"; line: 1, column: 4])
     at [Source: (byte[])"   {"; line: 1, column: 9]>}
    [2019-09-17T19:42:44,650][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"            \"account\": \"dev\"", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"            "account": "dev""; line: 1, column: 23]>}
    [2019-09-17T19:42:44,651][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"    }", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])"    }"; line: 1, column: 0])
     at [Source: (byte[])"    }"; line: 1, column: 6]>}
    [2019-09-17T19:42:44,651][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"            \"logStream\": \"testing/c78919c03baa4e1fa75adf06cf6d4139\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])"            "logStream": "testing/c78919c03baa4e1fa75adf06cf6d4139","; line: 1, column: 25]>}
6

You need to use a multiline codec to combine all the lines that comprise a single JSON object into one event.

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.