I have a Elastic search cluster with 4 nodes(each with 8 core cpu and 64 G RAM) . In my testing phase, I am trying to send max traffic to ES from a single client machine(8 core CPU) with {10, 15, 20, 30} concurrent client threads.
1st test
bulk size of 1000 with message size of 200bytes ->
When I increase the number of clients, I see that throughput increases till a point where CPU of each node goes to 90-100%. I assume that cluster is fully utilized at this point.
2nd test
bulk size of 1000 with message size of 4000bytes -
With 10 concurrent threads, I see that CPU of each node is just 50% . Since CPU is seen to be 50% idle , I tried to increase the concurrency. Even if I increase concurrent threads from 10 to 30, CPU of each node is still at 50%. I tried to check if there are any other bottlenecks (Disk, network) but I couldn't find any. Where can be the possible bottleneck in the system because of which throughput remains constant with any number of concurrent clients?