Facing Bulk queue rejection on index requests

[darthburrito (Ankit)]

Hi All,

Stack trace

Error: [es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$7@1b3a4fac on EsThreadPoolExecutor[bulk, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@334cd586[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 6853221]]
        at respond (/app/node_modules/elasticsearch/src/lib/transport.js:307:15)
        at checkRespForFailure (/app/node_modules/elasticsearch/src/lib/transport.js:266:7)
        at HttpConnector.<anonymous> (/app/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)
        at IncomingMessage.bound (/app/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)
        at emitNone (events.js:110:20)
        at IncomingMessage.emit (events.js:207:7)
        at endReadableNT (_stream_readable.js:1047:12)
        at _combinedTickCallback (internal/process/next_tick.js:102:11)
        at process._tickDomainCallback (internal/process/next_tick.js:198:9)

What am i doing

I am producing multiple concurrent client.index() requests from my node server to the ingest node

Cluster config

• 1 ingest node (m4.2xl)
• 3 dedicated master nodes ( t2.small)
• 6 data nodes ( 15 gb heap, 850Gb ssd nvme) ( i3.xlarge)

index settings

• shard_count = 6
• replica = 1

Question

why does the stack trace say the bulk queue is full when i am using the client.index() api ?
what is a good suggestion to handle these failures

[spinscale (Alexander Reelsen)]

if you send a single index request it gets converted to a bulk request internally. If you have many index requests going on, you should batch them together on the client side to not run into this problem.

Hope this helps.

--Alex

[zqc0512 (andy_zhou)]

change the bulk size with es (cpu cores.) and the queue line about bulk size
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-threadpool.html#modules-threadpool

[spinscale (Alexander Reelsen)]

Changing thread pool size or queue size is just a tradeoff by requiring more Elasticsearch memory to survive this issue longer, but does not solve fundamentlly the underlying problem.

[darthburrito (Ankit)]

thank you Alex, i was able to resolve this issue by using bulk api and throttling the number or requests on the application end