Logstash crash, ERROR Java heap space


(Alberto Gonzalez) #1

Logstash 5.0 crashed. Could this be insuficient memory? machine has 8GB RAM.

free -m

          total        used        free      shared  buff/cache   available

Mem: 7823 2388 1975 3 3459 5144
Swap: 1638 524 1114

these are last logged errors.

[2016-11-13T20:02:31,551][ERROR][org.logstash.beats.BeatsHandler] Exception: Java heap space
[root@docker02 ~]# docker logs 703ced116461 -f --tail 50
at io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:223) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:209) [netty-all-4.1.3.Final.jar:4.1.3.Final]
4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:478) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
[2016-11-13T20:00:35,463][WARN ][io.netty.channel.AbstractChannelHandlerContext] The exceptionCaught() event that was failed to submit was:
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:840) ~[netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:342) ~[netty-
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:478) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.safeExecute(SingleThreadEventExecutor.java:451) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:418) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:401) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:877) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144) [netty-all-4.1.3.Final.jar:4.1.3.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
[2016-11-13T20:00:35,468][WARN ][org.logstash.beats.Server] Channel initializer
[2016-11-13T20:00:35,469][WARN ][logstash.inputs.beats ] Error when creating a connection {:exception=>"event executor terminated"}

[2016-11-13T20:02:31,551][ERROR][org.logstash.beats.BeatsHandler] Exception: Java heap space


(João Duarte) #2

can you paste the configuration file?

have you tried opening the heap dump to check for big objects and their allocation point?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.