# Elasticsearch 7.1 have rediculously high disk read io (100%)

**URL:** <https://discuss.elastic.co/t/elasticsearch-7-1-have-rediculously-high-disk-read-io-100/225619>\
**Category:** Elasticsearch\
**Created:** [March 30, 2020, 7:08am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-have-rediculously-high-disk-read-io-100/225619 "2020-03-30T07:08:15Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![huntlyroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huntlyroad/32/65236_2.png) [@huntlyroad](https://discuss.elastic.co/u/huntlyroad)\
**Post date:** [March 30, 2020, 7:08am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-have-rediculously-high-disk-read-io-100/225619/1 "2020-03-30T07:08:15Z")

</div>

Hi,

We are seeing unexpected write I/O on elasticsearch data nodes.

this is what is captured by iotop:  
 ![Screenshot 2020-03-30 at 2.31.39 PM](https://us1.discourse-cdn.com/elastic/original/3X/0/1/016525b8953a7329a981371e831b26430809b6ae.png)

As you can see, the disk io is 100% all the time, following is captured detailed threads.

 ![Screenshot 2020-03-30 at 2.31.50 PM](https://us1.discourse-cdn.com/elastic/original/3X/5/a/5a5332c422d964347bc5e692bd44ca33fc31d85a.png)

Here is the output of Iostat

 ![Screenshot 2020-03-30 at 3.03.16 PM](https://us1.discourse-cdn.com/elastic/original/3X/0/9/09fd46c27b643695ffe23f5b1ae26df429877403.png)

What we did:

1. originally we had 3 master nodes, 5 data nodes, 3 coordinator nodes running in k8s version v1.16.6-beta.0 and docker version 18.9.7. All storage disks are premium ssds.
2. Recently we added 5 data nodes to this cluster.
3. We have a heavily read index which has 5 primary shards and each has 2 replicas.
4. It worked find previously however, because of the growth of the size, we are observing performance downgrade very slowly. that's why we are trying to scale up the data nodes.
5. as you can see, the newly added nodes have rediculously high read disk io. Top 20 threads are elasticsearch.....

Can any1 figure out what can be the potential reason?? thank you so much in advance.

---

<div class="post-metadata">

**Author:** ![huntlyroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huntlyroad/32/65236_2.png) [@huntlyroad](https://discuss.elastic.co/u/huntlyroad)\
**Post date:** [March 30, 2020, 8:07am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-have-rediculously-high-disk-read-io-100/225619/2 "2020-03-30T08:07:07Z")

</div>

We are observing very wired monitoring pattern

 ![Screenshot 2020-03-30 at 4.06.44 PM](https://us1.discourse-cdn.com/elastic/original/3X/9/a/9a9e5d1712f3cfd042d42d79baba0d94922bab60.png)

---

<div class="post-metadata">

**Author:** ![huntlyroad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/huntlyroad/32/65236_2.png) [@huntlyroad](https://discuss.elastic.co/u/huntlyroad)\
**Post date:** [March 30, 2020, 9:56am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-have-rediculously-high-disk-read-io-100/225619/3 "2020-03-30T09:56:00Z")

</div>

Looks like cache is not working, so it's reading from disk all the time. is it possible it's because I disabled THP and NUMA?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2020, 9:56am UTC](https://discuss.elastic.co/t/elasticsearch-7-1-have-rediculously-high-disk-read-io-100/225619/4 "2020-04-27T09:56:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.