How to increase thread pool

Hi,
I am having an error while I got 4 client.bulk program run at the same time:

Error: Request to Elasticsearch failed: "[es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$7@22f5ce2b on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@bfab946[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 44665]]"

I tried increase bulk thread pool by adding following into elasticsearch.yml

threadpool.bulk.type: fixed
threadpool.bulk.size: 25
threadpool.bulk.queue_size: 1000

But I got another error while I try to restart elasticsearch.service

[2017-09-20T17:44:53,724][INFO ][o.e.n.Node ] initializing ...
[2017-09-20T17:44:54,002][INFO ][o.e.e.NodeEnvironment ] [s9Tks9F] using [1] data paths, mounts [[/ (/dev/mapper/elasticsearch--vg-root)]], net usable_space [353.7gb], net total_space [481.7gb], spins? [possibly], types [ext4]
[2017-09-20T17:44:54,003][INFO ][o.e.e.NodeEnvironment ] [s9Tks9F] heap size [3.9gb], compressed ordinary object pointers [true]
[2017-09-20T17:44:54,170][INFO ][o.e.n.Node ] node name [s9Tks9F] derived from node ID [s9Tks9FwTPy9jE69qW-7mg]; set [node.name] to override
[2017-09-20T17:44:54,171][INFO ][o.e.n.Node ] version[5.5.2], pid[26617], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/4.4.0-93-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-09-20T17:44:54,171][INFO ][o.e.n.Node ] JVM arguments [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [aggs-matrix-stats]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [ingest-common]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [lang-expression]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [lang-groovy]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [lang-mustache]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [lang-painless]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [parent-join]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [percolator]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [reindex]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [transport-netty3]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [transport-netty4]
[2017-09-20T17:44:55,577][INFO ][o.e.p.PluginsService ] [s9Tks9F] no plugins loaded
[2017-09-20T17:44:56,763][ERROR][o.e.b.Bootstrap ] Exception
[2017-09-20T17:44:56,777][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]

Could I know how to solve this? Thank you very much.

Before you do this, I would suggest you read Any idea what these errors mean version 2.4.2

How many indices and shards are you indexing into? How many indexing threads does each loader use?

thank you!

I am doing same POC on the elasticsearch (single node), the elasticsearch I installed on the VM (10GB RAM, 4 processor).

4 incides are indexing at the same time,
index1, 20-40k data per 10s
index2, 40-60k data per 5mins
index3, 200 per 10s
index4, 1 per 10s

Exactly how are you loading this data? How large are your bulk requests? How large are your documents? How many connections are you indexing across?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.