Facing Bulk queue rejection on index requests

Elastic Stack Elasticsearch
1

Hi All,

Stack trace

Error: [es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$7@1b3a4fac on EsThreadPoolExecutor[bulk, queue capacity = 1000, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@334cd586[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 6853221]]
        at respond (/app/node_modules/elasticsearch/src/lib/transport.js:307:15)
        at checkRespForFailure (/app/node_modules/elasticsearch/src/lib/transport.js:266:7)
        at HttpConnector.<anonymous> (/app/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)
        at IncomingMessage.bound (/app/node_modules/elasticsearch/node_modules/lodash/dist/lodash.js:729:21)
        at emitNone (events.js:110:20)
        at IncomingMessage.emit (events.js:207:7)
        at endReadableNT (_stream_readable.js:1047:12)
        at _combinedTickCallback (internal/process/next_tick.js:102:11)
        at process._tickDomainCallback (internal/process/next_tick.js:198:9)

What am i doing

I am producing multiple concurrent client.index() requests from my node server to the ingest node

Cluster config

  • 1 ingest node (m4.2xl)
  • 3 dedicated master nodes ( t2.small)
  • 6 data nodes ( 15 gb heap, 850Gb ssd nvme) ( i3.xlarge)

index settings

  • shard_count = 6
  • replica = 1

Question :confused:

why does the stack trace say the bulk queue is full when i am using the client.index() api ?
what is a good suggestion to handle these failures

2

if you send a single index request it gets converted to a bulk request internally. If you have many index requests going on, you should batch them together on the client side to not run into this problem.

Hope this helps.

--Alex

1 Like
3

change the bulk size with es (cpu cores.) and the queue line about bulk size
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/modules-threadpool.html#modules-threadpool

4

Changing thread pool size or queue size is just a tradeoff by requiring more Elasticsearch memory to survive this issue longer, but does not solve fundamentlly the underlying problem.

1 Like
5

thank you Alex, i was able to resolve this issue by using bulk api and throttling the number or requests on the application end

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.