I am using grafana k6 stress tool to generate some load in elastic and sooner or later i always get this error es_rejected_execution_exception. All are search operations to get 30 docs.
-
On small indices of 20-30mb it works quite fine and respone time is 1sec until it gets to 500 virtual users and give the error.
-
On other indices that are 300mb it fails quite early just after a few requests and response time get to 14s and fails.
Same happen if we just do 3000 sequential requests that is not much instead of parallel users, i think this is very low load for elastic. After it get saturated elastic start rejecting queries and show this error in monitoris: Node xxx is reporting 10800 search rejections at day XXX
Elastic is 8.9.1 on k8s 1 node with 8GB and 4GB heap size
In monitoring i dont see any metric going very high.
Anyone can explain wthat is this error and why is caused? memory? cpu? any optimization to do?
Error:
[search_phase_execution_exception Caused by: es_rejected_execution_exception: rejected execution of TimedRunnable{original=org.elasticsearch.action.search.FetchSearchPhase$1@115599a3, creationTimeNanos=36855012795314, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = elasticsearch-es-master-0/search, queue capacity = 1000, task execution EWMA = 164ms, total task execution time = 18m, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@20706e70[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 144548]] Root causes: es_rejected_execution_exception: rejected execution of TimedRunnable{original=ActionRunnable#wrap[org.elasticsearch.search.SearchService$$Lambda$7936/0x00007f344d306668@33107600], creationTimeNanos=36855012420891, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = elasticsearch-es-master-0/search, queue capacity = 1000, task execution EWMA = 164ms, total task execution time = 18m, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@20706e70[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 144548]]: search_phase_execution_exception Caused by: es_rejected_execution_exception: rejected execution of TimedRunnable{original=org.elasticsearch.action.search.FetchSearchPhase$1@115599a3, creationTimeNanos=36855012795314, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = elasticsearch-es-master-0/search, queue capacity = 1000, task execution EWMA = 164ms, total task execution time = 18m, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@20706e70[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 144548]] Root causes: es_rejected_execution_exception: rejected execution of TimedRunnable{original=ActionRunnable#wrap[org.elasticsearch.search.SearchService$$Lambda$7936/0x00007f344d306668@33107600], creationTimeNanos=36855012420891, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = elasticsearch-es-master-0/search, queue capacity = 1000, task execution EWMA = 164ms, total task execution time = 18m, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@20706e70[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 144548]]]:
