# Is there any way to determine what ES doing caused abnormal IO?

**URL:** <https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring, docker\
**Created:** [March 24, 2022, 7:26am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527 "2022-03-24T07:26:28Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![mjikop1231](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mjikop1231/32/103447_2.png) [@mjikop1231](https://discuss.elastic.co/u/mjikop1231)\
**Post date:** [March 24, 2022, 7:26am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527/1 "2022-03-24T07:26:28Z")

</div>

I have a ES cluster with 3 master and 5 data node enabled rebalance and allocation. And in `/_nodes/.../stats` doesn't show any thing running in that nodes, and also no tasks in `_tasks`.  
But all data node has near 500 IOPS of write, and one of them have above 2000 IOPS of write.

So are there any way can I find how these IO happens?

* * *

Like I have same 4 data nodes in a cluster, and they have evenly 5k QPS, but one of them have near 2k write iops with other three have only 500 write iops. These nodes are hosted in completely same docker environment in different physicial server.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 28, 2022, 12:03am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527/2 "2022-03-28T00:03:58Z")

</div>

Welcome to our community! 😃

It could be merges happening automatically. Is it causing issues?

---

<div class="post-metadata">

**Author:** ![mjikop1231](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mjikop1231/32/103447_2.png) [@mjikop1231](https://discuss.elastic.co/u/mjikop1231)\
**Post date:** [March 28, 2022, 2:16am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527/3 "2022-03-28T02:16:51Z")

</div>

But I can't see any running merges in `stats`, and it makes search query more slower on this node.

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [March 28, 2022, 2:40am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527/4 "2022-03-28T02:40:50Z")

</div>

GET /\_nodes/hot\_threads

---

<div class="post-metadata">

**Author:** ![mjikop1231](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mjikop1231/32/103447_2.png) [@mjikop1231](https://discuss.elastic.co/u/mjikop1231)\
**Post date:** [March 28, 2022, 3:16am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527/5 "2022-03-28T03:16:01Z")

</div>

Seems nothing on that node.

```auto
::: {*}{CdzrK2KETQWMAfimJvKCoQ}{T5M3t5JPQPSOz0ybok7yWA}{*}{*}
   Hot threads at 2022-03-28T03:13:10.030Z, interval=500ms, busiestThreads=3, ignoreIdleThreads=true:

```

At this time, this node still have 1.2k write iops.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 28, 2022, 3:36am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527/6 "2022-03-28T03:36:49Z")

</div>

What does iostat or something similar show?

---

<div class="post-metadata">

**Author:** ![mjikop1231](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mjikop1231/32/103447_2.png) [@mjikop1231](https://discuss.elastic.co/u/mjikop1231)\
**Post date:** [March 28, 2022, 6:19am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527/7 "2022-03-28T06:19:38Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/5/0/50fec99cc0d16cb782f91917ba52b0e26f1123c6.png)  
Shows only this java process is causing write IO.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 25, 2022, 6:19am UTC](https://discuss.elastic.co/t/is-there-any-way-to-determine-what-es-doing-caused-abnormal-io/300527/8 "2022-04-25T06:19:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.