Logstash OOM from one host only

Elastic Stack Logstash
1

Hi,

I'm facing an issue I cant dig down. I have setup filebeat to ship logs from different hosts to logstash. I found that some logs were missing and I found an out of memory error in the logs:

    [2020-07-17T09:37:03,328][INFO ][org.logstash.beats.BeatsHandler][main][1a7b6b60de606e72fe759d281b5a4fa727bde4c285f36adad433cb80083db7d8] [local: 10.8.1.1:5044, remote: 10.4.5.1:47792] Handling exception: Direct buffer memory
    [2020-07-17T09:37:03,330][WARN ][io.netty.channel.DefaultChannelPipeline][main][1a7b6b60de606e72fe759d281b5a4fa727bde4c285f36adad433cb80083db7d8] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
    java.lang.OutOfMemoryError: Direct buffer memory
        	at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]

This happens only on one specifc host (which does not have a higher log volume than the others). I also increased heap from one to four GB of heap, but no change.

How to debug this?

Thanks for any help and suggestions.

Jan

2

It is running out of direct memory, not heap memory. Increasing the heap size may make the problem worse.

3

Is there any limiting factor for event sizes to cause this? The machine has plenty of ram - while this exception was thrown there were about 28GB free memory available.

4

Not that I know of. Direct buffers are native memory, but can only be freed when the Java objects that reference them are freed by garbage collection. Reducing the heap size to force GC to occur more often might help.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.