[SOLVED] How to manage HUGE log messages sent to ELK stack over-the-wire?

Elastic Stack Logstash
1

I've been using ELK for centralized logging service, and rsyslog as my shipper. Rsyslog wraps the log events in a JSON format.

However, some of our application logs are HUGE, e.g., stack traces, and when Logstash receives them, the log is truncated, resulting in _jsonparsefailure. I've tried sending the logs either via UDP/UDP6 and TCP/TCP6 to no avail.

How have some of you manage to solve this? I'm sure I'm not the only one in this predicament.

Thanks!

2

For rsysog users, see How to ship JSON logs via Rsyslog, especially the $MaxMessageSize setting. This along with the buffer_size setting solved my issues where large log events were being truncated.

3

Thanks for sharing this solution!