Hello,
We receive a lot of "rejected execution" messages in our ES logs.
E.g.:
2018-06-21 03:50:11,996][DEBUG][action.count ] [04c4efb4-9365-45d3-9c7b-162e3cbcc051] [logstash-2017.07.16][2], node[ac07Vu3JSl6vogmMAT4AYA], [P], s[STARTED]: failed to execute [[[]][], source[{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1529545511000,"to":1529545811000}}},{"fquery":{"query":{"query_string":{"query":"syslog_severity_code:([0 TO 3])"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"syslog_facility:(\"local1\")"}},"_cache":true}}]}}}}}]] org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@2de55b4a
I tried to monitor the search queue from the thread pool (for a few days).
I couldn't see anything abnormal (it does not look like the queue capacity is reached).
Indeed, the rejected counter is high.
"search" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 57558, "largest" : 2, "completed" : 164822}
Elasticsearch 1.6.0 is receiving data from Logstash 1.5.1.
I have 2 independent, separate ES deployments.
Deployment1
- 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
- 2 GB RAM
Deployment2
- 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
- 4 GB RAM
Both ES deployments have logs full of these messages.
Q1 (very important): does this message mean that some messages are not written in the ES database, i.e. some messages received by Logstash are lost?
Q2: Could someone offer a detailed explanation on the above DEBUG message from the elasticsearch server? (or we can further investigate the issue)
Thank you.
Regards,
Liviu