Hi,
I'm using a filebeat -> logstash -> logstash -> elasticsearch ->kibana stack. Logstash is version 6.7.1 and the others are also latest 6.x versions. On the downstream logstash, I saw this warning message when it keeps running for 2 days:
Apr 29 16:55:11 ncc1701a logstash[11969]: [2019-04-29T16:55:11,718][WARN ][logstash.inputs.beats ] Error when creating a connection {:exception=>"Direct buffer memory"}
Apr 29 16:55:11 ncc1701a logstash[11969]: [2019-04-29T16:55:11,718][WARN ][io.netty.channel.ChannelInitializer] Failed to initialize a channel. Closing: [id: 0xd42bc2fd, L:/192.168.71.35:5044 - R:/192.168.71.83:36436]
Apr 29 16:55:11 ncc1701a logstash[11969]: java.lang.OutOfMemoryError: Direct buffer memory
Apr 29 16:55:11 ncc1701a logstash[11969]: at java.nio.Bits.reserveMemory(Bits.java:175) ~[?:?]
Apr 29 16:55:11 ncc1701a logstash[11969]: at java.nio.DirectByteBuffer.(DirectByteBuffer.java:118) ~[?:?]
Apr 29 16:55:11 ncc1701a logstash[11969]: at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:317) ~[?:?]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.buffer.PoolArena$DirectArena.allocateDirect(PoolArena.java:764) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.buffer.PoolArena$DirectArena.newChunk(PoolArena.java:740) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.buffer.PoolArena.allocateNormal(PoolArena.java:244) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.buffer.PoolArena.allocate(PoolArena.java:226) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.buffer.PoolArena.allocate(PoolArena.java:146) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:324) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:185) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:176) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.PemX509Certificate.newBuffer(PemX509Certificate.java:152) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.PemX509Certificate.append(PemX509Certificate.java:134) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.PemX509Certificate.toPEM(PemX509Certificate.java:86) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.ReferenceCountedOpenSslContext.toBIO(ReferenceCountedOpenSslContext.java:772) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.ReferenceCountedOpenSslServerContext.newSessionContext(ReferenceCountedOpenSslServerContext.java:172) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.OpenSslServerContext.(OpenSslServerContext.java:352) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.OpenSslServerContext.(OpenSslServerContext.java:335) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.SslContext.newServerContextInternal(SslContext.java:422) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.handler.ssl.SslContextBuilder.build(SslContextBuilder.java:447) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at org.logstash.netty.SslSimpleBuilder.build(SslSimpleBuilder.java:128) ~[logstash-input-beats-5.1.8.jar:?]
Apr 29 16:55:11 ncc1701a logstash[11969]: at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:131) ~[logstash-input-beats-5.1.8.jar:?]
Apr 29 16:55:11 ncc1701a logstash[11969]: at org.logstash.beats.Server$BeatsInitializer.initChannel(Server.java:101) [logstash-input-beats-5.1.8.jar:?]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:115) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:107) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:637) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.channel.DefaultChannelPipeline.access$000(DefaultChannelPipeline.java:46) [netty-all-4.1.30.Final.jar:4.1.30.Final]
......
[netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:482) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Final]
Apr 29 16:55:11 ncc1701a logstash[11969]: at java.lang.Thread.run(Thread.java:834) [?:?]
After restart logstash, the message is gone. so it's definitely a logstash related issue. Does anybody have the same probelm? What should I do to avoid Direct buffer memory error? Thanks in advance! !
Regards