I'm using Logstash 5.2.2, Filebeat 5.2.2, Logstash-logback-encoder 4.7
I'm using net.logstash.logback.encoder.LogstashEncoder to write our application logs in JSON format. The logs themselves look fine.
Then I'm using ch.qos.logback.core.rolling.RollingFileAppender to roll logs with TimeBasedRollingPolicy writing to a new log each hour. There is no moving/renaming of logs, or zipping of old logs - each hour gets a completely fresh log file.
Because I am choosing to write logs in pure JSON format, I would expect parsing of those logs to be 100% reliable - since there is no need to worry about different log grok patterns, or different multi-line concerns. Filebeat just needs to take each single line of JSON from the log, and forward it through LS into ES.
But occasionally (maybe 100 times out of 5,000,000 lines of logs) I am seeing things like this:
{
"_index": "unmatched_raw_logs-2017.03.27",
"_type": "FOO_Bear_Processor",
"_id": "AVsNtOkNKJa3B0FS-Cpg",
"_score": null,
"_source": {
"@timestamp": "2017-03-27T03:00:08.665Z",
"module": "FOO_Bear_Processor",
"@version": "1",
"beat": {
"hostname": "xxxxxxxxxxxxxx",
"name": "internal_tomcat_beat",
"version": "5.2.2"
},
"source": "/FOO/logs/stash/FOO-bear-processor/FOO-bear-processor.log.2017-03-27_03",
"message": "el.breadcrumbId\":\"ID-xxxxxxxxxxxxxx-xxxxxxxx-xxxxx-com-43890-1490176523958-0-1360\",\"camel.routeId\":\"load-sheep-from-remote-system-route\"}",
"type": "FOO_Bear_Processor",
"tags": [
"_jsonparsefailure"
]
},
"fields": {
"@timestamp": [
1490583608665
]
},
"sort": [
1490583608665
]
}
Here, "el.breadcrumbId" is part of "camel.breadcrumbId" - the first characters of the line have been chopped off halfway through the word "camel", making the remaining characters become invalid JSON.
In this instance, the true original log line was approx 6100 characters long, and only the last 100 or so characters were fed into elastic. (I don't know whether the first 6000 chars were fed into elastic as a separate message, or whether they were just dropped completely)
The log in question (FOO-bear-processor.log.2017-03-27_03) was only 10 lines long (There were only ten lines of log written during that hour).
