JSON parse error, original data now in message field, even though its a valid json

[gopalkakularam]

This is the json log posting to logstash:

{
"@version" : 1,
"source_host" : "Gopals-MacBook-Pro-3.local",
"message" : "Loading source class org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration,class org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration,class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration,class org.springframework.boot.autoconfigure.context.PropertyPlaceholderAutoConfiguration, org.springframework.cloud.config.client.ConfigServiceBootstrapConfiguration,class org.springframework.cloud.config.client.DiscoveryClientConfigServiceBootstrapConfiguration",
"thread_name" : "main",
"@timestamp" : "2018-09-18T15:10:01.125-07:00",
"level" : "DEBUG",
"logger_name" : "org.springframework.boot.SpringApplication"
}

But in logstash logs, I could see below parsing error, even though its a valid json.

[2018-09-18T15:10:01,140][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"{"; line: 1, column: 1])

at [Source: (String)"{"; line: 1, column: 3]>, :data=>"{"}

[2018-09-18T15:10:01,141][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" "@version" : 1,"}

[2018-09-18T15:10:01,146][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" "source_host" : "Gopals-MacBook-Pro-3.local","}

[2018-09-18T15:10:01,147][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" "message" : "Loading source class org.springframework.cloud.bootstrap.config.PropertySourceBootstrapConfiguration,class org.springframework.cloud.bootstrap.encrypt.EncryptionBootstrapConfiguration,class org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration,class org.springframework.boot.autoconfigure.context.PropertyPlaceholderAutoConfiguration,class, org.springframework.cloud.config.client.ConfigServiceBootstrapConfiguration,class org.springframework.cloud.config.client.DiscoveryClientConfigServiceBootstrapConfiguration","}

[2018-09-18T15:10:01,148][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" "thread_name" : "main","}

[2018-09-18T15:10:01,149][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" "@timestamp" : "2018-09-18T15:10:01.125-07:00","}

[2018-09-18T15:10:01,150][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" "level" : "DEBUG","}

[2018-09-18T15:10:01,150][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: incompatible json object type=java.lang.String , only hash map or arrays are supported>, :data=>" "logger_name" : "org.springframework.boot.SpringApplication""}

[2018-09-18T15:10:01,151][WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected close marker '}': expected ']' (for root starting at [Source: (String)"}"; line: 1, column: 0])

at [Source: (String)"}"; line: 1, column: 2]>, :data=>"}"}

logstash-simple.conf:

input {

tcp {

port => 4560

codec => json_lines

}

}

output {

elasticsearch { hosts => ['localhost:9200'] }

}

output {

stdout {

codec => rubydebug

}

}

I have tried with codec json, json_line and filter with json.

[guyboertje]

You are trying to parse pretty printed JSON.

The default codec for the tcp input is lines, this will create an event for every line in the tcp payload as will the json and json_lines codecs. The json parser itself cannot parse one line from a pretty printed JSON string e.g. { or "source_host" : "Gopals-MacBook-Pro-3.local",

Use the plain codec in the tcp input to capture the full payload into one message field and then the json filter to parse the full payload message.