No events on Kibana

Elastic Stack Elasticsearch
1

We have a 3-node Elasticsearch cluster which receives events from different Filebeats sources. These events are collected by a Logstash server which sends them, through a Redis cache server, to another Logstash for filtering. Eventually the events are sent to the ES cluster.

Since several days we do not observe any index being logged into Kibana.

On the Logstash server that is supposed to receive events from Filebeats we observe these repeated stacktraces in the logs:

[2018-10-21T21:29:49,262][WARN ][io.netty.channel.AbstractChannelHandlerContext] Failed to submit an exceptionCaught() event.
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:267) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1030) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1326) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:183) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:167) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelUnregistered(DefaultChannelPipeline.java:835) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:830) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.18.Final.jar:4.1.18.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
[2018-10-21T21:29:52,476][WARN ][io.netty.channel.AbstractChannelHandlerContext] The exceptionCaught() event that was failed to submit was:
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.DefaultChannelPipeline.destroyUp(DefaultChannelPipeline.java:865) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.DefaultChannelPipeline.destroy(DefaultChannelPipeline.java:850) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.DefaultChannelPipeline.access$700(DefaultChannelPipeline.java:46) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelUnregistered(DefaultChannelPipeline.java:1341) ~[netty-all-4.1.18.Final.jar:4.1.18.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelUnregistered(AbstractChannelHandlerContext.java:181) [netty-all-4.1.18.Final.jar:4.1.18.Final]
(...)

What could we do to troubleshoot the issue?

2

What version of things are you on?

3

We use v6.3.2 for all components.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.