# Poor I/O on heavy aggregation **URL:** \ **Category:** Elasticsearch\ **Created:** [February 18, 2017, 11:01am UTC](https://discuss.elastic.co/t/poor-i-o-on-heavy-aggregation/75588 "2017-02-18T11:01:35Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [February 18, 2017, 11:01am UTC](https://discuss.elastic.co/t/poor-i-o-on-heavy-aggregation/75588/1 "2017-02-18T11:01:36Z")
Not sure if elasticsearch is involved, but, when doing a big aggregation (100millions doc, unique count on big cardinality fields, we use doc values), this: - takes 1 minute - consumes 50% CPU - RAM is stable, heap size is stable (no pike at all) - I/O goes max to 30 MB/s I am wondering if there is something that could limit performance? (we have SSD that should deliver 500MB/s, I am quite surprise). Thanks you, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 18, 2017, 11:58am UTC](https://discuss.elastic.co/t/poor-i-o-on-heavy-aggregation/75588/2 "2017-02-18T11:58:37Z")
How many shards are you aggregating across? How many nodes do you have in the cluster? Queries and aggregations are performed in a single thread per shard, but shards can be processed in parallel. If you therefore have a low number of shards per node that you are aggregating across and issue a single query, you can be limited by CPU even though it is not maxed out on the node. ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [February 18, 2017, 1:02pm UTC](https://discuss.elastic.co/t/poor-i-o-on-heavy-aggregation/75588/3 "2017-02-18T13:02:08Z")
Ha thanks you, didnt know that (1 thread per shard). I have a 3 data nodes cluster (and 2 LB nodes), of 7 indices with 3 shards each = 21 shards total. What metrics should I look at? Thanks you, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 18, 2017, 1:04pm UTC](https://discuss.elastic.co/t/poor-i-o-on-heavy-aggregation/75588/4 "2017-02-18T13:04:25Z")
Is that 21 primary shards? How many CPU cores does each node have? I would also recommend looking at iostat and see what the disk utilisation an await looks like. Read throughput on its own may not be the best measure as the load typically consists of a lot of random access reads. ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [February 18, 2017, 1:44pm UTC](https://discuss.elastic.co/t/poor-i-o-on-heavy-aggregation/75588/5 "2017-02-18T13:44:24Z")
Ya 21 primary shards, 1 replica = 42 shards total. We have SSD, so I am surprised to see maximum I/O read at 30MB/s: Zabbix screenshot =\> [http://b3.ms/Oq5nRGK5M0wB](http://b3.ms/Oq5nRGK5M0wB) As we use VMWare, I suspect a bad configuration from our sys-admin, should be that possible? Thanks you, ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 18, 2017, 7:43pm UTC](https://discuss.elastic.co/t/poor-i-o-on-heavy-aggregation/75588/6 "2017-02-18T19:43:28Z")
You may want to verify that the VMs have access to the resources to have assigned and are not over-provisioned with respect to CPU or RAM. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 18, 2017, 7:43pm UTC](https://discuss.elastic.co/t/poor-i-o-on-heavy-aggregation/75588/7 "2017-03-18T19:43:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.