Hey there, I am trying to increase the number of the thread pool size related to bulk but when I try nothing happens. The only way to do it is to use processor settings.
Can you explain it to me the reason?
Is there a reason you want to do this?
I'd suggest reading this topic for a solid refutation on why you think you might need to do this.
you question is obviously correct.
I am trying to tune DATA node's consumption performance, so I would like to understand if working on that aspect will give me any pros or cons.
my bulk queue is configured with an acceptable size (100), never really reached while the CPU usage is settle to 50%. So I would try to enforce resources utilisation
For high indexing workloads throughput is often limited by storage performance, but heap usage and garbage collection can also play a part. What type of storage are you using? Are you monitoring disk I/O and iowait? Are there any indications in the logs of long or frequeng garbage collection?
I have SSD on the machines.
Yes, I am monitoring disk I/O and iowait. Currently, ES is using 23% on CPU and I have iowait between 7% and 15%.
Generally, one time per day I saw these logs:
[2021-03-04 06:23:36,165][INFO ][monitor.jvm ] [5] [gc][young][322560][187173] duration [1.6s], collections [2]/[2.3s], total [1.6s]/[3.7h], memory [9.8gb]->[5.5gb]/[14.8gb], all_pools {[young] [4.4gb]->[72.1mb]/[6.2gb]}{[survivor] [540.4mb]->[37.5mb]/[800mb]}{[old] [4.9gb]->[5.4gb]/[7.8gb]}
Investigating into gc.log files I can see a plenty of the following messages:
2021-04-10T07:18:17.505+0000: 252292.573: Total time for which application threads were stopped: 0.0961067 seconds, Stopping threads took: 0.0002203 seconds
{Heap before GC invocations=67675 (full 111):
par new generation total 14745600K, used 13622715K [0x00007fe7bc000000, 0x00007feba4000000, 0x00007feba4000000)
eden space 13107200K, 100% used [0x00007fe7bc000000, 0x00007feadc000000, 0x00007feadc000000)
from space 1638400K, 31% used [0x00007feb40000000, 0x00007feb5f76ed30, 0x00007feba4000000)
to space 1638400K, 0% used [0x00007feadc000000, 0x00007feadc000000, 0x00007feb40000000)
concurrent mark-sweep generation total 16384000K, used 9231842K [0x00007feba4000000, 0x00007fef8c000000, 0x00007fef8c000000)
Metaspace used 54791K, capacity 55651K, committed 56280K, reserved 1099776K
class space used 5992K, capacity 6237K, committed 6400K, reserved 1048576K
2021-04-10T07:18:21.275+0000: 252296.343: [GC (Allocation Failure) 2021-04-10T07:18:21.275+0000: 252296.343: [ParNew: 13622715K->667518K(14745600K), 0.0937464 secs] 22854557K->9908152K(31129600K), 0.0940328 secs] [Times: user=1.10 sys=0.00, real=0.10 secs]
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.