How to adjust threadpools for elastic search clusters running on ECE

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

How do I configure settings like below for ece managed clusters.

thread_pool.bulk.queue_size: 100

I am using spark to ingest data into Elastic via the es-spark connector and am running into issues where I am seeing rejects on write. Also how does ECE assign cpu resources to clusters. Can we provide more cores to certain elastic clusters, I did not see that option anywhre during new cluster create.

T18/06/26 14:59:48 WARN TaskSetManager: Lost task 19.0 in stage 0.0 (TID 19, ip-xx-yy, executor 6): org.elasticsearch.hadoop.EsHadoopException: Could not write all entries for bulk operation [196/1000]. Error sample (first [5] error messages):
rejected execution of org.elasticsearch.transport.TransportService$7@34e59383 on EsThreadPoolExecutor[name = instance-0000000003/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@10558557[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 56516]]
rejected execution of org.elasticsearch.transport.TransportService$7@34e59383 on EsThreadPoolExecutor[name = instance-0000000003/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@10558557[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 56516]]
rejected execution of org.elasticsearch.transport.TransportService$7@34e59383 on EsThreadPoolExecutor[name = instance-0000000003/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@10558557[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 56516]]
rejected execution of org.elasticsearch.transport.TransportService$7@34e59383 on EsThreadPoolExecutor[name = instance-0000000003/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@10558557[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 56516]]
rejected execution of org.elasticsearch.transport.TransportService$7@34e59383 on EsThreadPoolExecutor[name = instance-0000000003/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@10558557[Running, pool size = 2, active threads = 2, queued tasks = 200, completed tasks = 56516]]
Bailing out...

2

Hi @rkalluri,

Setting the bulk thread pool can be done in the same way as you would set this in an on prem installation.
Select the relevant cluster and navigate to the Manage page, there you will be able to edit the elasticsearch.yaml file and add thread_pool.bulk.queue_size: 100, and then click on update.
After the activity progress will indicate that the change was implemented successfully you can verify it by navigating to the Console page and run GET /_nodes/thread_pool.

As for the other topic, CPU time is alloccated in the same ratio as the RAM. For example, in case a node with 16GB RAM is deployed on a host with 128GB RAM, the relative RAM assigned to the node will determine the CPU time; 16/128 - 12.5% meaning that node will be allocated with 12.5% of the total CPU time.

3

Thanks Zanbel that works.

4

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.