Failed to execute fetch phase SearchContextMissingException

Elastic Stack Elasticsearch
1

Hello,

We are constantly seeing below errors in elasticsearch log file. We want to know what could be the root cause of this problem and how to prevent these type of errors?

[2018-06-12T19:03:21,239][DEBUG][o.e.a.s.TransportSearchAction] [master-ip-x-x-x-x] [21240589] Failed to execute fetch phase
org.elasticsearch.transport.RemoteTransportException: [data-ip-x-x-x-x][x-x-x-x:9300][indices:data/read/search[phase/fetch/id]]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [21240589]
        at org.elasticsearch.search.SearchService.findContext(SearchService.java:442) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:409) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:402) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.action.search.SearchTransportService$12.messageReceived(SearchTransportService.java:399) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:258) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:307) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$authorizeAsync$5(ServerTransportFilter.java:191) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.maybeRun(AuthorizationUtils.java:127) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.setRunAsRoles(AuthorizationUtils.java:121) ~[?:?]
        at org.elasticsearch.xpack.security.authz.AuthorizationUtils$AsyncAuthorizer.authorize(AuthorizationUtils.java:109) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.authorizeAsync(ServerTransportFilter.java:193) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$2(ServerTransportFilter.java:162) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$0(AuthenticationService.java:192) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$2(AuthenticationService.java:212) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:224) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:190) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:147) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:118) ~[?:?]
        at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:145) ~[?:?]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:314) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1528) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) ~[elasticsearch-5.4.1.jar:5.4.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-5.4.1.jar:5.4.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]

Thanks!

2

Are you using scroll API?

3

@Igor_Motov No we are not using scroll API

4

Long running query or non-trivial network equipment (like load balancer, "smart"switches, etc ) between nodes? 3rd party plugins?

5

No third party plugins and also nodes are connected directly with each other. Not sure about long running searches but we see too many search rejection in thread_pool

6

Yes, this is possible when the cluster is overloaded with search traffic. Perhaps, it's time to increase the cluster capacity?

7

@Igor_Motov We are currently using hot warm architecture. We have 6 hot data nodes and 8 warm nodes. How can we overcome this problem without increasing cluster capacity? Can you provide us some guide how can we improve searching?

8

Sorry, I cannot really make any performance tuning suggestions based on a single stacktrace and the number of nodes. That's just not enough information. We would need to look at your cluster stats, analyze your queries, etc...

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.