Hi
I've got this message in logs of ES instance (it's part of a cluster, ES version is 6.2.3):
[2018-08-20T17:58:58,581][DEBUG][o.e.a.s.TransportSearchAction] [server01(master,data)] [logstash-index-2018.08.20][10], node[bqIEf6kyRjOoPUxY7X3qjQ], [R], s[STARTED], a[id=jzpb9x1dRUijMHCuUDiEGw]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[logstash--2018.08.20], indicesOptions=IndicesOptions[id=39, ignore_unavailable=true, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false], types=, routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=75, batchedReduceSize=512, preFilterShardSize=8, source={"size":0,"query":{"bool":{"filter":[{"range":{"@timestamp":{"from":"1534775321018","to":"1534777121018","include_lower":true,"include_upper":true,"format":"epoch_millis","boost":1.0}}},{"query_string":{"query":"<SOME_QUERY>","fields":,"type":"best_fields","default_operator":"or","max_determinized_states":10000,"enable_position_increments":true,"fuzziness":"AUTO","fuzzy_prefix_length":0,"fuzzy_max_expansions":50,"phrase_slop":0,"analyze_wildcard":true,"escape":false,"auto_generate_synonyms_phrase_query":true,"fuzzy_transpositions":true,"boost":1.0}}],"adjust_pure_negative":true,"boost":1.0}},"aggregations":{"2":{"date_histogram":{"field":"@timestamp","format":"epoch_millis","interval":"1m","offset":0,"order":{"_key":"asc"},"keyed":false,"min_doc_count":0,"extended_bounds":{"min":"1534775321018","max":"1534777121018"}}}}}}] lastShard [true]
org.elasticsearch.transport.RemoteTransportException: [server02(master,data)][<SOME_IP>:9300][indices:data/read/search[phase/query]]
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@19961b1c on QueueResizingEsThreadPoolExecutor[name = server02(master,data)/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 501.5ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@32017069[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 36101129]]
As I understand, it says that there is filled search queue on node server02, so search query was rejected.
According to this log, search queue capacity is 1000:
queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000
But I have such options in config files on both ES servers:
thread_pool.bulk.queue_size: 10000
thread_pool.search.queue_size: 20000
GET /_cat/thread_pool says that these options are applied.
In my opinion it's logging error. It should be fixed, because it's misleading and makes problem investigations more difficult.