I am running a 4 node cluster 2 data nodes 1.5 TB SSD each, 64gb Ram Each and Hexacore Processor,
1 Master node 16gb ram 4 core processor, 1 Client node 16gb ram 4 core processor. I am bulking index to http endpoint _bulk. I am currently only able to index 200k documents every 111 seconds. I am bulk indexing directly to 1 of my data nodes. This seems awfully slow. If I try to increase threads I start running into
es_rejected_execution_exception : rejected execution of org.elasticsearch.transport.TransportService$7@1736e37a on EsThreadPoolExecutor[name = ecluster01-dalc1-prod-data02/bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@5e8e7774[Running, pool size = 24, active threads = 24, queued tasks = 200, completed tasks = 374618]]
What can I be doing wrong? A single document is ~10k. I am running 10k batches of documents in my post.
Thank in advance