Hello,
I'm using Elasticsearch 6.7.2. I have hundreds of daily index with more or less 20 millions of documents each and one single shard per index.
I'm using scroll API to get some documents from this index. I've noticed this NullPointerException frequently on my database logs:
2020-01-07T20:28:06,686][DEBUG][o.e.a.s.TransportSearchScrollAction] [node_warm] [46307] Failed to execute query phase
org.elasticsearch.transport.RemoteTransportException: [node_warm][172.16.16.73:3310][indices:data/read/search[phase/query+fetch/scroll]]
Caused by: java.lang.NullPointerException
at org.elasticsearch.search.DefaultSearchContext.isCancelled(DefaultSearchContext.java:843) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:146) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:423) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$2(SearchService.java:517) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:381) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.2.jar:6.7.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_211]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
[2020-01-07T20:28:06,687][WARN ][r.suppressed ] [node_warm] path: /_search/scroll, params: {}
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.SearchScrollAsyncAction.onShardFailure(SearchScrollAsyncAction.java:269) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.action.search.SearchScrollAsyncAction$1.onFailure(SearchScrollAsyncAction.java:202) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:59) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:463) [elasticsearch-6.7.2.jar:6.7.2]
at com.floragunn.searchguard.transport.SearchGuardInterceptor$RestoringTransportResponseHandler.handleException(SearchGuardInterceptor.java:267) [search-guard-6-6.7.2-25.0.jar:6.7.2-25.0]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1114) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.processException(TransportService.java:1226) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.transport.TransportService$DirectResponseChannel.sendResponse(TransportService.java:1200) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:60) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.action.support.ChannelActionListener.onFailure(ChannelActionListener.java:56) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.SearchService$3.onFailure(SearchService.java:376) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.7.2.jar:6.7.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_211]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node_warm][172.16.16.73:3310][indices:data/read/search[phase/query+fetch/scroll]]
Caused by: java.lang.NullPointerException
at org.elasticsearch.search.DefaultSearchContext.isCancelled(DefaultSearchContext.java:843) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:146) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:423) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.SearchService.lambda$executeFetchPhase$2(SearchService.java:517) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.search.SearchService$3.doRun(SearchService.java:381) ~[elasticsearch-6.7.2.jar:6.7.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.7.2.jar:6.7.2]
... 6 more
After it, every time I try to use that scroll, I get this exception:
2020-01-07T20:28:07,065 [DEBUG][o.e.a.s.TransportSearchScrollAction] [node_warm] [46307] Failed to execute query phase
145128-org.elasticsearch.transport.RemoteTransportException: [node_warm][172.16.16.73:3310][indices:data/read/search[phase/query+fetch/scroll]]
145129-Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [46307]
Apparently, my scroll cease to exists. Important to say that I update the scroll_id after every request and my scroll living time are 1 day, much bigger than the time to perform another request. I found this topic NullPointerException in scroll with the same problem on another Elastic version, but without a solution or workaround.
Someone can help me?