# Unable to parse nested json using logstash json filter

**URL:** https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888
**Category:** Logstash
**Created:** [September 17, 2019, 10:17pm UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888 "2019-09-17T22:17:45Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![vj\_patnana](https://avatars.discourse-cdn.com/v4/letter/v/c57346/32.png) [@vj\_patnana](https://discuss.elastic.co/u/vj_patnana)
#### Post date: [September 17, 2019, 10:17pm UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888/1 "2019-09-17T22:17:45Z")

</div>

Hello All,

I am trying to parse logs which are in json format. I am trying to parse the **message** json field in the logs. I have tried using json plugin filter but it is not working as expected. Here is the log -

```
   {
    "id": "34984508843661791320106695906064879945445712328418263040",
    "timestamp": 1568759632569,
    "message": "{\"source_host\":\"ip-10-120-120-234.ec2.internal\",\"method\":\"newConnection\",\"level\":\"INFO\",\"appName\":\"testing\",\"message\":\"Attempting to open connection #1 to MySql\",\"mdc\":{},\"@timestamp\":\"2019-09-17T22:33:52.567Z\",\"file\":\"CachedConnectionProvider.java\",\"appGroup\":\"quality\",\"line_number\":\"88\",\"thread_name\":\"pool-17-thread-17\",\"@version\":1,\"logger_name\":\"io.regex.connect.jdbc.util.CachedConnectionProvider\",\"class\":\"io.regex.connect.jdbc.util.CachedConnectionProvider\"}",
    "aws": {
        "awslogs": {
            "logGroup": "testing",
            "logStream": "testing/c78919c03baa4e1fa75adf06cf6d4139",
            "owner": "2313245145",
            "account": "dev"
        },
        "tags": "awslogs"
    }
}

```

This is my logstash configuration -

```
    filter {
      json {
        source => "message"
      }
    }

```

Any help would be much appreciated.

Thanks in Advance

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 17, 2019, 11:19pm UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888/2 "2019-09-17T23:19:35Z")

</div>

If that is actually an example of what you are trying to parse then it does not have a [message] field, it has a [\_source][message] field. Normally I would assume that you are showing an example of the document in elasticsearch that results, but in that case the filter you show would parse it, so it suggests that assumption is incorrect.

Please clarify your question.

---

<div class="post-metadata">

### Author: ![vj\_patnana](https://avatars.discourse-cdn.com/v4/letter/v/c57346/32.png) [@vj\_patnana](https://discuss.elastic.co/u/vj_patnana)
#### Post date: [September 18, 2019, 12:00am UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888/3 "2019-09-18T00:00:07Z")

</div>

Sorry! My bad, I have posted the wrong sample log. Please have a look now, I have updated the log to the correct one. I am looking to parse **message** field.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 18, 2019, 12:12am UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888/4 "2019-09-18T00:12:15Z")

</div>

And even then that filter will parse that JSON. What exactly do you mean by "is not working as expected"? If you are getting an error message then please post that. If the parsing looks wrong then you need to be much more detailed.

You have to provide enough information for us to help you.

---

<div class="post-metadata">

### Author: ![vj\_patnana](https://avatars.discourse-cdn.com/v4/letter/v/c57346/32.png) [@vj\_patnana](https://discuss.elastic.co/u/vj_patnana)
#### Post date: [September 18, 2019, 12:57am UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888/5 "2019-09-18T00:57:34Z")

</div>

I was getting \_jsonparsefailure error. In the output section, I have enabled rubydebug and gave the above log event as stdin. Here is the error.log below. ( Couldn't attach complete log due to body limit to 7000 characters)

```
    }[2019-09-17T19:42:44,649][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"tags\": \"awslogs\"", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "tags": "awslogs""; line: 1, column: 16]>}
    [2019-09-17T19:42:44,649][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"id\": \"34984508843661791320106695906064879945445712328418263040\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "id": "34984508843661791320106695906064879945445712328418263040","; line: 1, column: 10]>}
    [2019-09-17T19:42:44,649][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"aws\": {", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "aws": {"; line: 1, column: 11]>}
    [2019-09-17T19:42:44,649][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"logGroup\": \"testing\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "logGroup": "testing","; line: 1, column: 24]>}
    [2019-09-17T19:42:44,649][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" },", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])" },"; line: 1, column: 0])
     at [Source: (byte[])" },"; line: 1, column: 10]>}
    [2019-09-17T19:42:44,649][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"message\": \"{\\\"source_host\\\":\\\"ip-10-120-120-234.ec2.internal\\\",\\\"method\\\":\\\"newConnection\\\",\\\"level\\\":\\\"INFO\\\",\\\"appName\\\":\\\"testing\\\",\\\"message\\\":\\\"Attempting to open connection #1 to MySql\\\",\\\"mdc\\\":{},\\\"@timestamp\\\":\\\"2019-09-17T22:33:52.567Z\\\",\\\"file\\\":\\\"CachedConnectionProvider.java\\\",\\\"appGroup\\\":\\\"quality\\\",\\\"line_number\\\":\\\"88\\\",\\\"thread_name\\\":\\\"pool-17-thread-17\\\",\\\"@version\\\":1,\\\"logger_name\\\":\\\"io.regex.connect.jdbc.util.CachedConnectionProvider\\\",\\\"class\\\":\\\"io.regex.connect.jdbc.util.CachedConnectionProvider\\\"}\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "message": "{\"source_host\":\"ip-10-120-120-234.ec2.internal\",\"method\":\"newConnection\",\"level\":\"INFO\",\"appName\":\"testing\",\"message\":\"Attempting to open connection #1 to MySql\",\"mdc\":{},\"@timestamp\":\"2019-09-17T22:33:52.567Z\",\"file\":\"CachedConnectionProvider.java\",\"appGroup\":\"quality\",\"line_number\":\"88\",\"thread_name\":\"pool-17-thread-17\",\"@version\":1,\"logger_name\":\"io.regex.connect.jdbc.util.CachedConnectionProvider\",\"class\":\"io.regex.connect.jd"[truncated 37 bytes]; line: 1, column: 15]>}
    [2019-09-17T19:42:44,650][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"owner\": \"2313245145\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "owner": "2313245145","; line: 1, column: 21]>}
    [2019-09-17T19:42:44,650][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"awslogs\": {", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "awslogs": {"; line: 1, column: 19]>}
    [2019-09-17T19:42:44,650][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"timestamp\": 1568759632569,", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "timestamp": 1568759632569,"; line: 1, column: 17]>}
    [2019-09-17T19:42:44,650][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" {", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])" {"; line: 1, column: 4])
     at [Source: (byte[])" {"; line: 1, column: 9]>}
    [2019-09-17T19:42:44,650][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"account\": \"dev\"", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "account": "dev""; line: 1, column: 23]>}
    [2019-09-17T19:42:44,651][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" }", :exception=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (byte[])" }"; line: 1, column: 0])
     at [Source: (byte[])" }"; line: 1, column: 6]>}
    [2019-09-17T19:42:44,651][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>" \"logStream\": \"testing/c78919c03baa4e1fa75adf06cf6d4139\",", :exception=>#<LogStash::Json::ParserError: Unexpected character (':' (code 58)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
     at [Source: (byte[])" "logStream": "testing/c78919c03baa4e1fa75adf06cf6d4139","; line: 1, column: 25]>}
```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 18, 2019, 12:24pm UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888/6 "2019-09-18T12:24:30Z")

</div>

You need to use a multiline codec to combine all the lines that comprise a single JSON object into one event.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 16, 2019, 12:24pm UTC](https://discuss.elastic.co/t/unable-to-parse-nested-json-using-logstash-json-filter/199888/7 "2019-10-16T12:24:33Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.