Idle Elastisearch cluster during Bulk loading

[jakesjohn]

I have a Elastic search cluster with 4 nodes(each with 8 core cpu and 64 G RAM) . In my testing phase, I am trying to send max traffic to ES from a single client machine(8 core CPU) with {10, 15, 20, 30} concurrent client threads.

1st test
bulk size of 1000 with message size of 200bytes ->
When I increase the number of clients, I see that throughput increases till a point where CPU of each node goes to 90-100%. I assume that cluster is fully utilized at this point.

2nd test
bulk size of 1000 with message size of 4000bytes -
With 10 concurrent threads, I see that CPU of each node is just 50% . Since CPU is seen to be 50% idle , I tried to increase the concurrency. Even if I increase concurrent threads from 10 to 30, CPU of each node is still at 50%. I tried to check if there are any other bottlenecks (Disk, network) but I couldn't find any. Where can be the possible bottleneck in the system because of which throughput remains constant with any number of concurrent clients?

[jakesjohn]

Or in other words, Bulk throughput is remaining constant even if we increase the number of concurrent client threads with system resources( CPU, disk and network) not fully utilized. Where can be the bottleneck?