We're experiencing the following DEBUG & WARN log messages and experiencing an issue where sporadically our searches appear to not be returning when these messages are seen. They all seem to begin with the "Received ban for the parent" message. Other times the same searches return fine from the same index.
We recently added 2 nodes (from 1) and upgraded to 7.6.2 in hopes of resolving this issue, but had no luck. I was hoping that someone has seen or knows what could be causing this. Any insight would be greatly appreciated, we're relatively new to using Elasticsearch.
DEBUG o.e.a.a.c.n.t.c.TransportCancelTasksAction - Received ban for the parent [YcCWN54BR_acDGPTgocR8g:1847161] on the node [YcCWN54BR_acDGPTgocR8g], reason: [by user request]
DEBUG o.e.a.s.TransportSearchAction - [250739] Failed to execute fetch phase
DEBUG o.e.a.s.TransportSearchAction - [250741] Failed to execute fetch phase
DEBUG o.e.a.s.TransportSearchAction - [250740] Failed to execute fetch phase
DEBUG o.e.a.s.TransportSearchAction - All shards failed for phase: [query]
DEBUG o.e.a.a.c.n.t.c.TransportCancelTasksAction - Sending remove ban for tasks with the parent [YcCWN54BR_acDGPTgocR8g:1847161] to the node [YcCWN54BR_acDGPTgocR8g]
DEBUG o.e.a.a.c.n.t.c.TransportCancelTasksAction - Removing ban for the parent [YcCWN54BR_acDGPTgocR8g:1847161] on the node [YcCWN54BR_acDGPTgocR8g]
WARN r.suppressed - path: /XXXXX/_search, params: {index=XXXXX}
Full Log messages:
[2020-04-01T15:32:27,684][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [TESTNODE] Received ban for the parent [YcCWN54BR_acDGPTgocR8g:1847161] on the node [YcCWN54BR_acDGPTgocR8g], reason: [by user request]
[2020-04-01T15:32:27,684][DEBUG][o.e.a.s.TransportSearchAction] [TESTNODE] [250739] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [TESTNODE][XX.XXX.XX..XX:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:149) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$6(SearchService.java:502) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$runAsync$2(SearchService.java:344) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:45) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
[2020-04-01T15:32:27,684][DEBUG][o.e.a.s.TransportSearchAction] [TESTNODE] [250741] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [TESTNODE][XX.XXX.XX..XX:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:149) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$6(SearchService.java:502) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$runAsync$2(SearchService.java:344) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:45) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
[2020-04-01T15:32:27,684][DEBUG][o.e.a.s.TransportSearchAction] [TESTNODE] [250740] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [TESTNODE][XX.XXX.XX..XX:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:149) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$6(SearchService.java:502) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$runAsync$2(SearchService.java:344) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:45) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
[2020-04-01T15:32:27,684][DEBUG][o.e.a.s.TransportSearchAction] [TESTNODE] All shards failed for phase: [query]
org.elasticsearch.tasks.TaskCancelledException: cancelled
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:149) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$6(SearchService.java:502) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$runAsync$2(SearchService.java:344) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:45) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
[2020-04-01T15:32:27,700][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [TESTNODE] Sending remove ban for tasks with the parent [YcCWN54BR_acDGPTgocR8g:1847161] to the node [YcCWN54BR_acDGPTgocR8g]
[2020-04-01T15:32:27,700][DEBUG][o.e.a.a.c.n.t.c.TransportCancelTasksAction] [TESTNODE] Removing ban for the parent [YcCWN54BR_acDGPTgocR8g:1847161] on the node [YcCWN54BR_acDGPTgocR8g]
[2020-04-01T15:32:27,700][WARN ][r.suppressed ] [TESTNODE] path: /test_index/_search, params: {index=test_index}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:314) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executeNextPhase(AbstractSearchAsyncAction.java:139) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.search.FetchSearchPhase.moveToNextPhase(FetchSearchPhase.java:210) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.search.FetchSearchPhase.lambda$innerRun$2(FetchSearchPhase.java:104) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.search.CountedCollector.countDown(CountedCollector.java:53) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.search.CountedCollector.onFailure(CountedCollector.java:76) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.search.FetchSearchPhase$2.onFailure(FetchSearchPhase.java:177) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:424) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1120) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1229) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1203) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:60) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) [elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.4.2.jar:7.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
Caused by: org.elasticsearch.tasks.TaskCancelledException: cancelled
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:149) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$6(SearchService.java:502) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.search.SearchService.lambda$runAsync$2(SearchService.java:344) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.action.ActionRunnable$1.doRun(ActionRunnable.java:45) ~[elasticsearch-7.4.2.jar:7.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.4.2.jar:7.4.2]
... 6 more