My problem is pretty similar to this one. We have a dedicated monitoring cluster with 2 nodes that we upgraded to 5.6.9 from 5.4 a month ago. Recently, we have not been able to load the "Advanced Tab" for any individual node. Every time we try we get a 503 search_phase_execution_exception. After looking into the error logs I found that this is due to us blowing through the thread limit for searches.
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.action.search.FetchSearchPhase$1@6fed4e8b on EsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@39018d64[Running, pool size = 7, active threads = 7, queued tasks = 1003, completed tasks = 111115001]]
This monitoring cluster is not heavily used and when I pull up the stats for searching on it you can see it really is not doing anything until we try to load one of the monitoring pages. Both node search stats look similar to this one.
- Timeframe I am looking at is 1 hour
- All indexes on the cluster are green
- We have tried to close old indexes in an effort to solve this issue but it has not worked. The cluster currently has 280 open .monitoring-es and .monitoring-kibana indexes
Unlike our main cluster which I have control over how we handle searching I don't know much about how Kibana executes it searches so I am not sure what the best way to go about fixing this is.
Thanks in advanced for any help!