# Elasticsearch 6.3 CPU load & Disk I/O increased **URL:** \ **Category:** Elasticsearch\ **Created:** [September 12, 2018, 12:26pm UTC](https://discuss.elastic.co/t/elasticsearch-6-3-cpu-load-disk-i-o-increased/148318 "2018-09-12T12:26:59Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Itay\_Bittan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itay_bittan/32/23527_2.png) [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Post date:** [September 12, 2018, 12:26pm UTC](https://discuss.elastic.co/t/elasticsearch-6-3-cpu-load-disk-i-o-increased/148318/1 "2018-09-12T12:26:59Z")
Hi, I'm working on upgrading our Elasticsearch cluster from 1.4.3 to 6.3.0 Both old & new cluster has 3 master nodes, 4 client (coordinating) nodes and 11 data nodes (total 18 nodes). Cluster runs on AWS. Instance types doesn't changed except data nodes which improved! (16 cores instead of 8) and we are working with ephemeral instead of EBS to improve performance. Both cluster has the same data: ~1700 indices ~ 7000 shards ~550M docs 830 GB total both cluster handle up to 1000 requests per second. The request are identical (we are duplicating traffic to the new cluster). We did the necessary changes to query so they will work on ES6, nothing else. somehow, the new cluster perform much more I/O. new cluster (iostat): Device: tps kB\_read/s kB\_wrtn/s kB\_read kB\_wrtn xvda 4.00 0.00 27.50 0 27 xvdb 3325.00 182836.00 48.00 182836 48 xvdc 3317.00 182888.00 40.00 182888 40 md127 25986.00 368484.00 88.00 368484 88 old cluster (iostat): Device: tps kB\_read/s kB\_wrtn/s kB\_read kB\_wrtn xvda 0.00 0.00 0.00 0 0 xvdb 732.00 14980.00 1532.00 14980 1532 In addition, New cluster load average is much higher than the old one and I see higher load each time the Disk I/O is high. Any idea what can cause this behavior? Thanks ---
**Author:** ![Itay\_Bittan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itay_bittan/32/23527_2.png) [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Post date:** [September 12, 2018, 12:31pm UTC](https://discuss.elastic.co/t/elasticsearch-6-3-cpu-load-disk-i-o-increased/148318/2 "2018-09-12T12:31:03Z")
adding more information: **new cluster index settings:** s\_8770202\_1536039001363: { - settings: { - index: { - routing: { - search: { - number\_of\_shards: "1", - provided\_name: "s\_8770202\_1536039001363", - creation\_date: "1536039040790", - analysis: { - number\_of\_replicas: "6", - uuid: "qPWQuhTDSmWwK2cCzRHgaw", - version: { } **old cluster index example:** s\_8770202\_1527741946002: { - settings: { - index: { - creation\_date: "1527742287597", - routing: { - uuid: "2UiGcKk-StuN2YKrAtQvfg", - analysis: { - cache: { - number\_of\_replicas: "6", - number\_of\_shards: "1", - latest\_valid\_version: "1527464715828", - version: { } ---
**Author:** ![Itay\_Bittan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itay_bittan/32/23527_2.png) [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Post date:** [September 12, 2018, 12:38pm UTC](https://discuss.elastic.co/t/elasticsearch-6-3-cpu-load-disk-i-o-increased/148318/3 "2018-09-12T12:38:04Z")
Elasticsearch.yml comparison: **1.4** cluster.name: rcom-es-cluster cluster.routing.allocation.awareness.attributes: availability\_zone node.name: data-srv-rcom-es-node-5fc6f11 node.master: false node.data: true node.zone: SERVER node.availability\_zone: us-east-1a node.max\_local\_storage\_nodes: 1 index.mapper.dynamic: true action.auto\_create\_index: true action.disable\_delete\_all\_indices: true path.conf: /etc/elasticsearch path.data: /storage/elasticsearch path.logs: /var/log/elasticsearch bootstrap.mlockall: true http.port: 9200 http.cors.enabled: true http.cors.allow-origin: /.\*/ http.cors.allow-credentials: true gateway.recover\_after\_nodes: 1 gateway.recover\_after\_time: 5m gateway.expected\_nodes: 2 cluster.routing.allocation.node\_concurrent\_recoveries: 8 discovery.type: ec2 discovery.zen.minimum\_master\_nodes: 1 discovery.zen.ping.multicast.enabled: false cloud.node.auto\_attributes: true cloud.aws.access\_key: cloud.aws.secret\_key: cloud.aws.region: us-east-1 discovery.ec2.host\_type: private\_ip discovery.ec2.availability\_zones: us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f discovery.ec2.tag.rcom: true action.destructive\_requires\_name: true index.load\_fixed\_bitset\_filters\_eagerly: false index.refresh\_interval: 30s index.search.slowlog.threshold.fetch.warn: 500ms index.search.slowlog.threshold.query.warn: 500ms indices.cache.filter.size: 10% indices.fielddata.cache.size: 50% indices.memory.index\_buffer\_size: 30% indices.store.throttle.type: none monitor.jvm.gc.ConcurrentMarkSweep.info: 5s monitor.jvm.gc.ConcurrentMarkSweep.warn: 10s monitor.jvm.gc.ParNew.info: 700ms monitor.jvm.gc.ParNew.warn: 1s threadpool.bulk.queue\_size: 3000 threadpool.search.queue\_size: 6000 **6.3** cluster.name: rcom-es6-cluster node.name: data-server-25bd51e path.data: "/storage/elasticsearch" path.logs: "/var/log/elasticsearch" network.host: 0.0.0.0 discovery.zen.hosts\_provider: ec2 discovery.zen.minimum\_master\_nodes: 1 discovery.ec2.tag.dy\_cluster: rcom-es6-cluster discovery.ec2.host\_type: private\_ip action.auto\_create\_index: true action.destructive\_requires\_name: true cloud.node.auto\_attributes: true cluster.routing.allocation.node\_concurrent\_recoveries: 8 discovery.ec2.availability\_zones: us-east-1a,us-east-1b,us-east-1c,us-east-1d,us-east-1e,us-east-1f xpack.security.enabled: false xpack.ml.enabled: false node.ml: false xpack.monitoring.collection.enabled: true http.cors.allow-credentials: true http.cors.allow-origin: "/.\*/" http.cors.enabled: true http.port: 9200 indices.fielddata.cache.size: 50% indices.memory.index\_buffer\_size: 30% node.max\_local\_storage\_nodes: 1 node.master: false node.data: true node.ingest: false search.remote.connect: false node.attr.zone: SERVER cluster.routing.allocation.awareness.attributes: zone node.attr.availability\_zone: us-east-1 ---
**Author:** ![jaddison](https://avatars.discourse-cdn.com/v4/letter/j/e5b9ba/32.png) [@jaddison](https://discuss.elastic.co/u/jaddison)\ **Post date:** [September 12, 2018, 3:51pm UTC](https://discuss.elastic.co/t/elasticsearch-6-3-cpu-load-disk-i-o-increased/148318/4 "2018-09-12T15:51:06Z")
I'm not an expert scaling out elasticsearch nodes, but might the fact that you have `number_of_replicas` set to 0 in the new cluster be a problem? This is preventing the 'even' distribution of queries across replicas. ---
**Author:** ![Itay\_Bittan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itay_bittan/32/23527_2.png) [@Itay\_Bittan](https://discuss.elastic.co/u/Itay_Bittan)\ **Post date:** [September 12, 2018, 4:46pm UTC](https://discuss.elastic.co/t/elasticsearch-6-3-cpu-load-disk-i-o-increased/148318/5 "2018-09-12T16:46:31Z")
thanks @jaddison, it was a bad example and it's not the usual case. most of my indices has replication factor 6 (I have 8 "serving" data nodes). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 10, 2018, 4:46pm UTC](https://discuss.elastic.co/t/elasticsearch-6-3-cpu-load-disk-i-o-increased/148318/6 "2018-10-10T16:46:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.