Elasticsearch Error "Event loop shutdown"

[nguacon90 (Quang Minh)]

Hi All, I'm using elastic search with information as following:

    {
    "name": "3KOuo1W",
    "cluster_name": "common-cluster",
    "cluster_uuid": "H9IcVILhROqC76KMneIxlA",
    "version": {
    "number": "5.5.3",
    "build_hash": "9305a5e",
    "build_date": "2017-09-07T15:56:59.599Z",
    "build_snapshot": false,
    "lucene_version": "6.6.0"
    },
    "tagline": "You Know, for Search"
    }

Sometime, Elasticsearch throw error:

> {"log":"[2020-08-17T08:31:49,173][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down?\n","stream":"stdout","time":"2020-08-17T08:31:49.180194296Z"}

I've been checking the server, the eslasticsearch using over 100% CPU.
Anyone can help me to resolve this error?
Thanks.

[warkolm (Mark Walkom)]

Welcome to our community!

5.X is EOL, you should really upgrade as a matter of urgency.
Otherwise, is there more in your logs?

[nguacon90 (Quang Minh)]

Thanks for your reply.
Currently it's hard to upgrade new version because the system run over 3 years, and serve over 1 million users, also having many data with old structure that not compatible with new version.
I can provide more logs. Can you help me?

[2020-08-17T08:31:49,173][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:821)
io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:327)
io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:320)
io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:746)
io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:163)
io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93)
io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28)
org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:396)
org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:94)
org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:1122) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1195) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1165) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:67) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:61) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:60) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:111) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:68) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.support.HandledTransportAction$TransportHandler$1.onResponse(HandledTransportAction.java:64) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:291) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.AbstractSearchAsyncAction.onResponse(AbstractSearchAsyncAction.java:49) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.FetchSearchPhase$3.run(FetchSearchPhase.java:215) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:143) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:137) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.ExpandSearchPhase.run(ExpandSearchPhase.java:119) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:143) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:137) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:208) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:105) ~[elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.FetchSearchPhase.innerRun(FetchSearchPhase.java:117) [elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.FetchSearchPhase.access$000(FetchSearchPhase.java:45) [elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.action.search.FetchSearchPhase$1.doRun(FetchSearchPhase.java:87) [elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.5.3.jar:5.5.3]
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.5.3.jar:5.5.3]
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_141]
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_141]
java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

[warkolm (Mark Walkom)]

Is that the only log entry, there's nothing before or after that?

[nguacon90 (Quang Minh)]

That's true. Nothing before or after this log, therefore i can't know what's wrong