Logstash version is 6.4.0, Filebeat is 6.4.2.
Today, we are receiving the following exceptions continuously on Logstash (several of this appear):
[2019-12-25T14:17:07,113][INFO ][org.logstash.beats.BeatsHandler] [local: 172.17.0.2:5044, remote: 10.0.1.2:51896] Handling exception: java.lang.IndexOutOfBoundsException: writerIndex(2147483408) + minWritableBytes(512) exceeds maxCapacity(2147483647): PooledUnsafeDirectByteBuf(ridx: 0, widx: 2147483408, cap: 2147483647)
This is typically followed, after few mins, with: (several of them appear)
[2019-12-25T15:32:16,429][INFO ][org.logstash.beats.BeatsHandler] [local: 172.17.0.2:5044, remote: 10.0.1.2:52820] Handling exception: failed to allocate 1556086784 byte(s) of direct memory (used: 24146608119, max: 25700073472)
After this, Logstash does not seem to process any logs and will need to be restarted.
The IP 10.0.1.2 has filebeat and has lots of logs (4-5GB of logs), with some of the log messages are greater than 1MB and max upto 15MB (each log line size).
What is the probable cause and how to fix this?
Edit: Initially the logstash heap size was 7GB (has been so for the past year), we increased it upto 24GB. But still Logstash throws up messages, for not enough memory.