Rejected execution log messages

Hello,

We receive a lot of "rejected execution" messages in our ES logs.
E.g.:

 2018-06-21 03:50:11,996][DEBUG][action.count ] [04c4efb4-9365-45d3-9c7b-162e3cbcc051] [logstash-2017.07.16][2], node[ac07Vu3JSl6vogmMAT4AYA], [P], s[STARTED]: failed to execute [[[]][], source[{"query":{"filtered":{"query":{"bool":{"should":[{"query_string":{"query":"*"}}]}},"filter":{"bool":{"must":[{"range":{"@timestamp":{"from":1529545511000,"to":1529545811000}}},{"fquery":{"query":{"query_string":{"query":"syslog_severity_code:([0 TO 3])"}},"_cache":true}},{"fquery":{"query":{"query_string":{"query":"syslog_facility:(\"local1\")"}},"_cache":true}}]}}}}}]]
    org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1@2de55b4a

I tried to monitor the search queue from the thread pool (for a few days).
I couldn't see anything abnormal (it does not look like the queue capacity is reached).
Indeed, the rejected counter is high.
"search" : { "threads" : 2, "queue" : 0, "active" : 0, "rejected" : 57558, "largest" : 2, "completed" : 164822}

Elasticsearch 1.6.0 is receiving data from Logstash 1.5.1.
I have 2 independent, separate ES deployments.
Deployment1

  • 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
  • 2 GB RAM

Deployment2

  • 1 CPU core (Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz)
  • 4 GB RAM

Both ES deployments have logs full of these messages.

Q1 (very important): does this message mean that some messages are not written in the ES database, i.e. some messages received by Logstash are lost?

Q2: Could someone offer a detailed explanation on the above DEBUG message from the elasticsearch server? (or we can further investigate the issue)

Thank you.
Regards,
Liviu

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.