Loogstash is not parsing the message correctly to kibana

[Jonesthomas]

hi,my sample log {"LogLevel":"ERROR","LogMsg":"{\"itemId\":0,\"module\":\"/curie/encounter\",\"action\":\"/addToken\",\"errorMessage\":\"java.lang.RuntimeException: org.apache.ibatis.exceptions.PersistenceException: \\n### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\\n### The error may exist in EncounterMapper.xml\\n### The error may involve EncounterMapper.checkTokenExist-Inline\\n### The error occurred while setting parameters\\n### SQL: SELECT COUNT(*) FROM token WHERE appointmentId \\u003d ?\\n### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\",\"parameter\":\"java.lang.RuntimeException: java.lang.RuntimeException: org.apache.ibatis.exceptions.PersistenceException: \\n### Error querying database. Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.\\n### The error may exist in EncounterMapper.xml\\n### The error may involve EncounterMapper.checkTokenExist-Inline\\n### The error occurred while setting parameters\\n### SQL: SELECT COUNT(*) FROM token WHERE appointmentId \\u003d ?\\n### Cause: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: The last packet successfully received from the server was 134,717,933 milliseconds ago. The last packet sent successfully to the server was 134,717,972 milliseconds ago. is longer than the server configured value of \\u0027wait_timeout\\u0027. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property \\u0027autoReconnect\\u003dtrue\\u0027 to avoid this problem.systems.ellora.core.api.encounter.domain.EncounterBuilder.addToken(EncounterBuilder.java:995)\"}","Time":"2017-11-20_09:12:21.042"}
my filter plugin:
filter { json { source => "message" skip_on_invalid_json => true } json { source => "LogLevel" skip_on_invalid_json => true } json { source => "LogMsg" skip_on_invalid_json => true } if [LogLevel] == "INFO" { drop { remove_field => [ "LogLevel" ] } } mutate { add_field => { "ErrorMsg" => "%{errorMessage}" } } truncate { fields => "ErrorMsg" length_bytes => 1000 } }

ErrorMsg field is not working correctly

[Jonesthomas]

anyone

[Dheeraj_Gupta (Dheeraj)]

You have only one field (LogMsg) which is being stored as a string. errorMessage is actually in LogMsg.

You will need to reference it like so

"ErrorMsg" => "%{[LogMsg][errorMessage]}"

Or it may be better to explicitly rename fields so that they are not under LogMsg

mutate {
    rename => {
        "[LogMsg][errorMessage]" => "errorMessage"
        "[LogMsg][module]" => "module"
         ... so on
    }
}

In which case your original filter should work.

[Jonesthomas]

tried this "ErrorMsg" => "%{[LogMsg][errorMessage]}" my filter:
filter { json { source => "message" skip_on_invalid_json => true } json { source => "LogLevel" skip_on_invalid_json => true } json { source => "LogMsg" skip_on_invalid_json => true } if [LogLevel] == "INFO" { drop { remove_field => [ "LogLevel" ] } } mutate { add_field => { "ErrorMsg" => "%{[LogMsg][errorMessage]}" } } truncate { fields => "ErrorMsg" length_bytes => 500 } }

[Dheeraj_Gupta (Dheeraj)]

I suspect your LogMsg field is being treated as a large string rather than a JSON containing more fields.

[Jonesthomas]

but its a valid json ,what i supposed to do now..

[Badger]

I don't know if this will help, since in your first screenshot errorMessage is set to 2452/2116, which indicates that LogMsg did get parsed. However, in 6.0, a json filter will not parse LogMsg in the form you posted into the original message. That can be fixed by stripping out the newlines embedded in it with

  mutate { gsub => [ "LogMsg", "
", "" ] }

And no, there is no escaping the newline, just put the open and close quotes on different lines.