Logstash can't handle large messages

[rzwarts (RaymondZ)]

Hi all,

I'm having an issue with logstash and large syslog messages. I am using logstash 7.8 and a json template to get the files sent from rsyslog to logstash. But some messages are too long and completely crash the server (well if the server only had 1 partition).

Can anyone help me here ?

Jun 24 15:20:33 cyg7803 logstash[810]: [2020-06-24T15:20:33,653][ERROR][logstash.codecs.json     ][main][40e3dc6b71c528b0a2b69d0a8873f8ad2b8cbd7efa95a1fb7d8baa0d71d8f031] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected end-of-input in character escape sequence
Jun 24 15:20:33 cyg7803 logstash[810]:  at [Source: (String)"{"@timestamp":"2020-06-24T15:20:33.652747+02:00","@version":"1","message":"  at [Source: (String)\"{\"@timestamp\":\"2020-06-24T15:20:33.651830+02:00\",\"@version\":\"1\",\"message\":\"  at [Source: (String)\\\"{\\\"@timestamp\\\":\\\"2020-06-24T15:20:33.650965+02:00\\\",\\\"@version\\\":\\\"1\\\",\\\"message\\\":\\\"  at [Source: (String)\\\\\\\"{\\\\\\\"@timestamp\\\\\\\":\\\\\\\"2020-06-24T15:20:33.648040+02:00\\\\\\\",\\\\\\\"@version\\\\\\\":\\\\\\\"1\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"[truncated 7596 chars]; line: 1, column: 16193]>, :data=>"{\"@timestamp\":\"2020-06-24T15:20:33.652747+02:00\",\"@version\":\"1\",\"message\":\"  at [Source: (String)\\\"{\\\"@timestamp\\\":\\\"2020-06-24T15:20:33.651830+02:00\\\",\\\"@version\\\":\\\"1\\\",\\\"message\\\":\\\"  at [Source: (String)\\\\\\\"{\\\\\\\"@timestamp\\\\\\\":\\\\\\\"2020-06-24T15:20:33.650965+02:00\\\\\\\",\\\\\\\"@version\\\\\\\":\\\\\\\"1\\\\\\\",\\\\\\\"message\\\\\\\":\\\\\\\"  at [Source: (String)\\\\\\\\\\\\\\\"{\\\\\\\\\\\\\\\"@timestamp\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"2020-06-24T15:20:33.648040+02:00\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"@version\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"1\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\"message\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\"  at [Source: (String)\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"{\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"@timestamp\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"2020-06-24T15:20:33.647184+02:00\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"@version\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"1\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\",\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"message\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\":\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"[truncated 7596 chars];

[Badger]

What is your input configuration?

[rzwarts (RaymondZ)]

It pretty basic. I removed the IPs but input is the rsyslog json forward and output is an elastic search host

input {
  udp {
    host => "x.x.x.x"
    port => 10514
    codec => "json"
    type => "rsyslog"
  }
}
# This is an empty filter block.  You can later add other filters here to further process
# your log lines
filter { }
# This output block will send all events of type "rsyslog" to Elasticsearch at the configured
# host and port into daily indices of the pattern, "rsyslog-YYYY.MM.DD"
output {
  if [type] == "rsyslog" {
    elasticsearch {
      hosts => [ "x.x.x.x:9200" ]
    }
  }
}

[rzwarts (RaymondZ)]

I actually changed my rsyslog config as I might have run into the 8K limit in message size that is the default.

Rsyslog documentation

$MaxMessageSize <size_nbr>, default 8k - allows to specify maximum supported message size (both for sending and receiving). The default should be sufficient for almost all cases. Do not set this below 1k, as it would cause interoperability problems with other syslog implementations.