I'm running tests to size up a cluster and get the best tuning parameters for my requirements - Index documents at the rate of 10k per second (each document being ~2k in size) across 10 indices, each having 3 shards each with async replication of 1
There is a 4 node cluster of c3.xlarge instances ES 1.6.0 with data stored on 2 instance store SSD drives. I'm generating bulk indexing requests at from a storm cluster of 3 nodes using NodeClient. So far I have been able to reach upto 2.5k but I seem to have reached a point where I can't figure out the bottleneck.
At a rate of 2.5k per second the cpu utilization on the nodes is only around 25%. But any increase in document indexing rate results in EsRejectedExecutionException (TransportShardReplicationOperationAction$PrimaryPhase).
The logs seem to suggest a lot of GC activity, but I can't seem to get better of this cluster at this point of time. Any help will be appreciated.