Hi Guys,
I am trying to call multiple(300) elasticsearch search request Async and I got the following error. It seems like some of my requests being rejected. Anyone experience this before.
I am still new to Elasticsearch, please correct me if any wrong. Thanks
2019-08-30T16:21:32,588][DEBUG][o.e.a.s.TransportSearchAction] [Windows-PC] [data][4], node[tYdfN-D-REye4n7029yCQw], [P], s[STARTED], a[id=z4waoy-VRwuXqO3x6XOypQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[log], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=, routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"query":{"query_string":{"query":"data.id:3","fields":,"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":false,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [Windows-PC][127.0.0.1:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@7a92c500 on QueueResizingEsThreadPoolExecutor[name = WIN10-PC/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 2000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 41.4ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@538a0253[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 5766]]
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:?]
at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:98) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$4(SearchService.java:1061) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.shard.IndexShard.awaitShardSearchActive(IndexShard.java:3024) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$5(SearchService.java:1060) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService.rewriteShardRequest(SearchService.java:1076) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:354) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$7(SearchTransportService.java:352) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:687) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TransportService.access$000(TransportService.java:73) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:125) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:633) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:543) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:587) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:578) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:137) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:54) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$4(InitialSearchPhase.java:259) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.InitialSearchPhase.performPhaseOnShard(InitialSearchPhase.java:294) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.InitialSearchPhase.run(InitialSearchPhase.java:158) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:166) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:126) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:514) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:399) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$3(TransportSearchAction.java:211) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) ...