# Unable to fully utilize resources in Elasticsearch cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [July 25, 2015, 2:09am UTC](https://discuss.elastic.co/t/unable-to-fully-utilize-resources-in-elasticsearch-cluster/26279 "2015-07-25T02:09:30Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [July 25, 2015, 2:09am UTC](https://discuss.elastic.co/t/unable-to-fully-utilize-resources-in-elasticsearch-cluster/26279/1 "2015-07-25T02:09:30Z")
Hi, I'm running tests to size up a cluster and get the best tuning parameters for my requirements - Index documents at the rate of 10k per second (each document being ~2k in size) across 10 indices, each having 3 shards each with async replication of 1 There is a 4 node cluster of c3.xlarge instances ES 1.6.0 with data stored on 2 instance store SSD drives. I'm generating bulk indexing requests at from a storm cluster of 3 nodes using NodeClient. So far I have been able to reach upto 2.5k but I seem to have reached a point where I can't figure out the bottleneck. At a rate of 2.5k per second the cpu utilization on the nodes is only around 25%. But any increase in document indexing rate results in EsRejectedExecutionException (TransportShardReplicationOperationAction$PrimaryPhase). The logs seem to suggest a lot of GC activity, but I can't seem to get better of this cluster at this point of time. Any help will be appreciated. ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [July 25, 2015, 2:31am UTC](https://discuss.elastic.co/t/unable-to-fully-utilize-resources-in-elasticsearch-cluster/26279/2 "2015-07-25T02:31:47Z")
Here are some more info to troubleshoot: [iostat of all nodes](https://gist.github.com/Srinathc/1e0c0c306a33f8febc0f#file-iostat_all) [hot threads - node 1](https://gist.github.com/Srinathc/1e0c0c306a33f8febc0f#file-node1-hot-threads-txt) [hot threads - node 2](https://gist.github.com/Srinathc/1e0c0c306a33f8febc0f#file-node2-hot-threads-txt) [hot threads - node 3](https://gist.github.com/Srinathc/1e0c0c306a33f8febc0f#file-node3-hot-threads-txt) [hot threads - node 4](https://gist.github.com/Srinathc/1e0c0c306a33f8febc0f#file-node4-hot-threads-txt) Let me know if anything else would help in giving pointers. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 25, 2015, 7:03am UTC](https://discuss.elastic.co/t/unable-to-fully-utilize-resources-in-elasticsearch-cluster/26279/3 "2015-07-25T07:03:37Z")
Can you please share some additional information about your cluster configuration, e.g. heap size and any non-default settings? What bulk size are you using? What type oaf data are you indexing? In addition to the indexing, how much querying is going on at the same time? ---
**Author:** ![Srinath\_C](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/srinath_c/32/48806_2.png) [@Srinath\_C](https://discuss.elastic.co/u/Srinath_C)\ **Post date:** [July 25, 2015, 10:35am UTC](https://discuss.elastic.co/t/unable-to-fully-utilize-resources-in-elasticsearch-cluster/26279/4 "2015-07-25T10:35:33Z")
Hi Christian, Thanks for looking into this. I have uploaded the [elasticsearch.yml](https://gist.github.com/Srinathc/1e0c0c306a33f8febc0f#file-elasticsearch-yml). Here are the JVM args used: -server -Djava.net.preferIPv4Stack=true -Xms4479m -Xmx4479m -Xss256k -XX:NewRatio=1 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError There are three NodeClients using BulkRequestBuilder and configured to do bulk index for 10000 or in 10 seconds whichever is earlier. Data is being pushed in the form of JSON, (most of them have inner documents too). The current tests are mostly checking the indexing rate hence queries are very minimal ...(\<10s per minute). But we do plan to have around 10 queries (mostly aggregations over 1 hour or 1 day) per second later on. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:59pm UTC](https://discuss.elastic.co/t/unable-to-fully-utilize-resources-in-elasticsearch-cluster/26279/5 "2017-07-05T23:59:01Z")