How to increase thread pool

Elastic Stack Elasticsearch
1

Hi,
I am having an error while I got 4 client.bulk program run at the same time:

Error: Request to Elasticsearch failed: "[es_rejected_execution_exception] rejected execution of org.elasticsearch.transport.TransportService$7@22f5ce2b on EsThreadPoolExecutor[bulk, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@bfab946[Running, pool size = 4, active threads = 4, queued tasks = 200, completed tasks = 44665]]"

I tried increase bulk thread pool by adding following into elasticsearch.yml

threadpool.bulk.type: fixed
threadpool.bulk.size: 25
threadpool.bulk.queue_size: 1000

But I got another error while I try to restart elasticsearch.service

[2017-09-20T17:44:53,724][INFO ][o.e.n.Node ] initializing ...
[2017-09-20T17:44:54,002][INFO ][o.e.e.NodeEnvironment ] [s9Tks9F] using [1] data paths, mounts [[/ (/dev/mapper/elasticsearch--vg-root)]], net usable_space [353.7gb], net total_space [481.7gb], spins? [possibly], types [ext4]
[2017-09-20T17:44:54,003][INFO ][o.e.e.NodeEnvironment ] [s9Tks9F] heap size [3.9gb], compressed ordinary object pointers [true]
[2017-09-20T17:44:54,170][INFO ][o.e.n.Node ] node name [s9Tks9F] derived from node ID [s9Tks9FwTPy9jE69qW-7mg]; set [node.name] to override
[2017-09-20T17:44:54,171][INFO ][o.e.n.Node ] version[5.5.2], pid[26617], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/4.4.0-93-generic/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_131/25.131-b11]
[2017-09-20T17:44:54,171][INFO ][o.e.n.Node ] JVM arguments [-Xms4g, -Xmx4g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [aggs-matrix-stats]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [ingest-common]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [lang-expression]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [lang-groovy]
[2017-09-20T17:44:55,575][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [lang-mustache]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [lang-painless]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [parent-join]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [percolator]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [reindex]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [transport-netty3]
[2017-09-20T17:44:55,576][INFO ][o.e.p.PluginsService ] [s9Tks9F] loaded module [transport-netty4]
[2017-09-20T17:44:55,577][INFO ][o.e.p.PluginsService ] [s9Tks9F] no plugins loaded
[2017-09-20T17:44:56,763][ERROR][o.e.b.Bootstrap ] Exception
[2017-09-20T17:44:56,777][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]

Could I know how to solve this? Thank you very much.

2

Before you do this, I would suggest you read Any idea what these errors mean version 2.4.2

3

How many indices and shards are you indexing into? How many indexing threads does each loader use?

4

thank you!

5

I am doing same POC on the elasticsearch (single node), the elasticsearch I installed on the VM (10GB RAM, 4 processor).

4 incides are indexing at the same time,
index1, 20-40k data per 10s
index2, 40-60k data per 5mins
index3, 200 per 10s
index4, 1 per 10s

6

Exactly how are you loading this data? How large are your bulk requests? How large are your documents? How many connections are you indexing across?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.