# Loogstash is not parsing the message correctly to kibana

**URL:** <https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344>\
**Category:** Logstash\
**Created:** [November 20, 2017, 9:05am UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344 "2017-11-20T09:05:32Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [November 20, 2017, 9:05am UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344/1 "2017-11-20T09:05:32Z")

</div>

hi,my sample log `{"LogLevel":"ERROR","LogMsg":"{\"itemId\":0,\"module\":\"/curie/encounter\",\"action\":\"/addToken\",\"errorMessage\":\"java.lang.RuntimeException: org.apache.ibatis.exceptions.PersistenceException: \\n### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\\n### The error may exist in EncounterMapper.xml\\n### The error may involve EncounterMapper.checkTokenExist-Inline\\n### The error occurred while setting parameters\\n### SQL: SELECT COUNT(*) FROM token WHERE appointmentId \\u003d ?\\n### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\",\"parameter\":\"java.lang.RuntimeException: java.lang.RuntimeException: org.apache.ibatis.exceptions.PersistenceException: \\n### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\\n### The error may exist in EncounterMapper.xml\\n### The error may involve EncounterMapper.checkTokenExist-Inline\\n### The error occurred while setting parameters\\n### SQL: SELECT COUNT(*) FROM token WHERE appointmentId \\u003d ?\\n### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.systems.ellora.core.api.encounter.domain.EncounterBuilder.addToken(EncounterBuilder.java:995)\"}","Time":"2017-11-20_09:12:21.042"}`  
my filter plugin:  
`filter { json { source => "message" skip_on_invalid_json => true } json { source => "LogLevel" skip_on_invalid_json => true } json { source => "LogMsg" skip_on_invalid_json => true } if [LogLevel] == "INFO" { drop { remove_field => ["LogLevel"] } } mutate { add_field => { "ErrorMsg" => "%{errorMessage}" } } truncate { fields => "ErrorMsg" length_bytes => 1000 } }`  
 ![Untitled](https://us1.discourse-cdn.com/elastic/original/3X/8/7/872afdf8810d114cd9e9ecb8374734d004625c91.png)  
ErrorMsg field is not working correctly

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [November 22, 2017, 4:49am UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344/2 "2017-11-22T04:49:48Z")

</div>

anyone

---

<div class="post-metadata">

**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\
**Post date:** [November 22, 2017, 7:00am UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344/3 "2017-11-22T07:00:38Z")

</div>

You have only one field (`LogMsg`) which is being stored as a string. `errorMessage` is actually _in_ `LogMsg`.

You will need to reference it like so

`"ErrorMsg" => "%{[LogMsg][errorMessage]}"`

Or it may be better to explicitly rename fields so that they are not under `LogMsg`

```
mutate {
    rename => {
        "[LogMsg][errorMessage]" => "errorMessage"
        "[LogMsg][module]" => "module"
         ... so on
    }
}

```

In which case your original filter should work.

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [November 22, 2017, 7:30am UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344/4 "2017-11-22T07:30:58Z")

</div>

tried this _"ErrorMsg" =\> "%{[LogMsg][errorMessage]}"_ my filter:  
`filter { json { source => "message" skip_on_invalid_json => true } json { source => "LogLevel" skip_on_invalid_json => true } json { source => "LogMsg" skip_on_invalid_json => true } if [LogLevel] == "INFO" { drop { remove_field => ["LogLevel"] } } mutate { add_field => { "ErrorMsg" => "%{[LogMsg][errorMessage]}" } } truncate { fields => "ErrorMsg" length_bytes => 500 } }`

 ![Untitled](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a29307ae2eb1416a7aee256abda479ba448f232c.png)

---

<div class="post-metadata">

**Author:** ![Dheeraj\_Gupta](https://avatars.discourse-cdn.com/v4/letter/d/49beb7/32.png) [@Dheeraj\_Gupta](https://discuss.elastic.co/u/Dheeraj_Gupta)\
**Post date:** [November 22, 2017, 7:40am UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344/5 "2017-11-22T07:40:31Z")

</div>

I suspect your `LogMsg` field is being treated as a large string rather than a JSON containing more fields.

---

<div class="post-metadata">

**Author:** ![Jonesthomas](https://avatars.discourse-cdn.com/v4/letter/j/f4b2a3/32.png) [@Jonesthomas](https://discuss.elastic.co/u/Jonesthomas)\
**Post date:** [November 22, 2017, 7:42am UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344/6 "2017-11-22T07:42:36Z")

</div>

but its a valid json ,what i supposed to do now..☹

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 22, 2017, 3:58pm UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344/7 "2017-11-22T15:58:33Z")

</div>

I don't know if this will help, since in your first screenshot errorMessage is set to 2452/2116, which indicates that LogMsg did get parsed. However, in 6.0, a json filter will not parse LogMsg in the form you posted into the original message. That can be fixed by stripping out the newlines embedded in it with

```
mutate { gsub => [ "LogMsg", "
", "" ] }
```

And no, there is no escaping the newline, just put the open and close quotes on different lines.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 20, 2017, 3:59pm UTC](https://discuss.elastic.co/t/loogstash-is-not-parsing-the-message-correctly-to-kibana/108344/8 "2017-12-20T15:59:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.