Recently, Elastic Cloud has added the ability to also ship logs to a remote monitoring cluster. Enabling that feature worked great, however, I'm seeing parsing issues in the elasticsearch.server dataset.
Two subsequent log documents sourced from the same node (let's call it instance-44) and the same JSON log file (/app/logs/fb503f***_server.json) are not parsed the same way. All fields are pretty much identical except for the message field. In the first document, it looks like this:
"@timestamp" : "2020-10-08T05:16:55.630Z",
"message" : "duration [999ms], collections [1]/[1.3s], total [999ms]/[2s], memory [9.1gb]->[9.2gb]/[29gb], all_pools {[young] [712mb]->[16mb]/[0b]}{[old] [6.5gb]->[8.8gb]/[29gb]}{[survivor] [1.8gb]->[472mb]/[0b]}",
While in the second document, like this:
"@timestamp" : "2020-10-08T05:16:55.639Z",
"message" : """{"type": "server", "timestamp": "2020-10-08T05:16:55,639Z", "level": "WARN", "component": "o.e.m.j.JvmGcMonitorService", "cluster.name": "fb503f***", "node.name": "instance-0000000044", "message": "[gc][3308] overhead, spent [999ms] collecting in the last [1.3s]", "cluster.uuid": "_TWTJE05***", "node.id": "fug7snda***" }""",
Instead the message field should be:
"message": "[gc][3308] overhead, spent [999ms] collecting in the last [1.3s]"
All the fields from that stringified JSON are actually in the end document, so it looks like the message field was indeed parsed, however, it was not replaced with the actual message [gc][3308] overhead...
When looking those two logs directly in the ES Cloud Console, they both look right, though:
Also worth noting that the index called elastic-cloud-logs-7-2020.10.08-000001 has no index.final_pipeline or index.default_pipeline set up that could potentially fail to do the parsing job, the issue must be happening on the Filebeat side that is installed within Elastic Cloud.