Hello,
Lately my logstash instance has been throwing the following exceptions intermittently. I cannot find a solution anywhere. All the posts in this forum with the string "writerIndex" with similar issues are left unanswered.
Here is the java stack (reverse order of time):
Aug 25 12:11:29 logstash.example.com logstash[11784]: ... 9 more
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at org.logstash.beats.BeatsParser.decode(BeatsParser.java:181) ~[logstash-input-beats-6.2.3.jar:?]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at org.logstash.beats.BeatsParser.inflateCompressedFrame(BeatsParser.java:218) ~[logstash-input-beats-6.2.3.jar:?]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at org.logstash.beats.BeatsParser.decompressImpl(BeatsParser.java:231) ~[logstash-input-beats-6.2.3.jar:?]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.buffer.AbstractByteBuf.readBytes(AbstractByteBuf.java:964) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.buffer.PooledUnsafeDirectByteBuf.getBytes(PooledUnsafeDirectByteBuf.java:142) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.buffer.UnsafeByteBufUtil.getBytes(UnsafeByteBufUtil.java:602) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.buffer.UnsafeByteBufUtil.getBytes(UnsafeByteBufUtil.java:615) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at java.util.zip.InflaterOutputStream.write(InflaterOutputStream.java:255) ~[?:?]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.buffer.ByteBufOutputStream.write(ByteBufOutputStream.java:67) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1073) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.buffer.AbstractByteBuf.ensureWritable(AbstractByteBuf.java:280) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.buffer.AbstractByteBuf.ensureWritable0(AbstractByteBuf.java:294) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: Caused by: java.lang.IndexOutOfBoundsException: writerIndex(2147483267) + minWritableBytes(512) exceeds maxCapacity(2147483647): PooledUnsafeDirectByteBuf(ridx: 0, widx: 2147483267, cap: 2147483647)
Aug 25 12:11:29 logstash.example.com logstash[11784]: at java.lang.Thread.run(Thread.java:829) [?:?]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:370) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:61) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:477) ~[netty-all-4.1.65.Final.jar:4.1.65.Final]
Aug 25 12:11:29 logstash.example.com logstash[11784]: io.netty.handler.codec.DecoderException: java.lang.IndexOutOfBoundsException: writerIndex(2147483267) + minWritableBytes(512) exceeds maxCapacity(2147483647): PooledUnsafeDirectByteBuf(ridx: 0, widx: 2147483267, cap: 2147483647)
Aug 25 12:11:29 logstash.example.com logstash[11784]: [2022-08-25T12:11:29,589][WARN ][io.netty.channel.DefaultChannelPipeline][main][32670a15682f17f0b0f99dc060e96c7e2287bba72ac084e0031aa0639c2f1132] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
Aug 25 12:11:29 logstash.example.com logstash[11784]: [2022-08-25T12:11:29,583][INFO ][org.logstash.beats.BeatsHandler][main][32670a15682f17f0b0f99dc060e96c7e2287bba72ac084e0031aa0639c2f1132] [local: 172.31.20.93:5044, remote: 167.74.98.248:51015] Handling exception: io.netty.handler.codec.DecoderException: java.lang.IndexOutOfBoundsException: writerIndex(2147483267) + minWritableBytes(512) exceeds maxCapacity(2147483647): PooledUnsafeDirectByteBuf(ridx: 0, widx: 2147483267, cap: 2147483647) (caused by: java.lang.IndexOutOfBoundsException: writerIndex(2147483267) + minWritableBytes(512) exceeds maxCapacity(2147483647): PooledUnsafeDirectByteBuf(ridx: 0, widx: 2147483267, cap: 2147483647))
Aug 25 12:02:23 logstash.example.com logstash[11784]: [2022-08-25T12:02:23,127][INFO ][org.logstash.beats.Server][main][32670a15682f17f0b0f99dc060e96c7e2287bba72ac084e0031aa0639c2f1132] Starting server on port: 5044
Aug 25 12:02:23 logstash.example.com logstash[11784]: [2022-08-25T12:02:23,065][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:".monitoring-logstash", :main], :non_running_pipelines=>[]}
Aug 25 12:02:22 logstash.example.com logstash[11784]: [2022-08-25T12:02:22,903][INFO ][logstash.javapipeline ][main] Pipeline started {"pipeline.id"=>"main"}
Aug 25 12:02:22 logstash.example.com logstash[11784]: [2022-08-25T12:02:22,858][INFO ][logstash.inputs.beats ][main] Starting input listener {:address=>"0.0.0.0:5044"}
Thank you for any assistance.
Joey