Broken JSON (Rsyslog -> Logstash) - Message breaking in wrong place

Elastic Stack Logstash
1

Hi guys,

I have a setup where my rsyslog log messages are sent to Logstash in JSON format.

My template in rsyslog is this:

template(name="json-template"
  type="list") {
    constant(value="{")
      constant(value="\"@timestamp\":\"")     property(name="timereported" dateFormat="rfc3339")
      constant(value="\",\"@version\":\"1")
      constant(value="\",\"message\":\"")     property(name="msg" format="json")
      constant(value="\",\"sysloghost\":\"")  property(name="hostname")
      constant(value="\",\"severity\":\"")    property(name="syslogseverity-text")
      constant(value="\",\"facility\":\"")    property(name="syslogfacility-text")
      constant(value="\",\"programname\":\"") property(name="programname")
      constant(value="\",\"procid\":\"")      property(name="procid")
    constant(value="\"}\n")
}

This used to works in later 6.X versions. But after upgrading to 7.x (not sure if that's the reason, but coincided) it's now broken: Some of our log messages are being tagged with _jsonparsefailure. Inspecting the original message that's now stored on "message" field, it always looks like a broken json.

Here some example of messages that failed:

programname":"CRON","procid":"14896"}

{"@timestamp":"2019-11-04T10:39:01.503132-03:00","@version":"1","message":" (www-data) CMD (php /var/www/artisan schedule:run >> /dev/null 2>&1)","sysloghost":"fe67d4888ff3","severity":"info","facility":"cron","programname":"CRON","procid":"14897"}

0:36:36.358225-03:00","@version":"1","message":" custom.INFO: Triggered Aaaaa\\Bbbbbbbbb\\EmailMessageCreated [] []","sysloghost":"fe67d4888ff3","severity":"info","facility":"user","programname":"changedforsafety","procid":"12611"}

y":"cron","programname":"CRON","procid":"14675"}

Note that the message is always broken, something like if the logstash (or rsyslog) was not parsing the json from the start.

Can someone help me with that?

2

Searching again I found this topic: JSON data split up (_jsonparseerror) in logstash-7.2.0-1. Works fine in logstash-7.1.1-1

Looks like my problem is similar. I just changed my codec from json to json_lines and will wait a few hours to check if the problem disappear.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.