Logstash config parameter to handle large messages from rsyslog server

Elastic Stack Logstash
1

Hello Team

Developers use on the device a conf with a max size of 65MB because of bigger core dumps.

$MaxMessageSize 65536k

What will be the logstash configuration parameter to handle these big-size messages? our current conf is as below

data:
      logstash.conf: |
        input {
            tcp {
                port                  => 5140
                type                  => "device-logs"
                ssl_enable            => "true"
                tcp_keep_alive        => "true"
                ssl_cert              => "/etc/pki/logstash/syslog-listener.crt"
                ssl_key               => "/etc/pki/logstash/syslog-listener.key"
                ssl_extra_chain_certs => ["/etc/pki/logstash/syslog-listener-ca.crt"]
                ssl_certificate_authorities => ["/etc/pki/logstash/syslog-listener-ca.crt"]
            }
        }
2

Is the parameter max_message_size: 70MiB will help?

3

There is no such setting for the tcp input, the only settings available are the ones described in the documentation.

From the documentation there is no setting related to the message size.

Are you having any issues? It is not clear.

4

Hello @leandrojmp

As per the update from the developer

From time to time, so messages (mainly when a core dump occurs) devices stop sending message logstash and only with a reset of journalctl, we can send messages again.
By sending a bigger message (around 1 MB) I was able to reproduce this behavior.
5

@leandrojmp Also, any possible solution or workaround to overcome this issue? I mean, to handle the big logs/messages?

6

From time to time, so messages (mainly when a core dump occurs) devices stop sending message logstash and only with a reset of journalctl, we can send messages again.
By sending a bigger message (around 1 MB) I was able to reproduce this behavior.

How are you sending your messages? You mentioned journald, then you have a rsyslog configuration in your original post, and you have logstash in the receiving side, it is not clear what is the data flow here.

The message you shared mentiones that the device stop sending message, Logstash is on the receiving side.

It seems that the issue starts on the sending side.

65 MB is too big, it will probably require some network tunning regarding the buffer size and many other configurations, but these is not related to Logstash, you may need to look anywhere else on how to tune the operating system tcp configuration on both your client and server.

7

Thank you for the update @leandrojmp

Yes, rsyslog is the sender and logsatsh is the receiver.