# One of the nodes in cluster has much higher number of write operations than other nodes

**URL:** <https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425>\
**Category:** Elasticsearch\
**Created:** [December 17, 2015, 2:50am UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425 "2015-12-17T02:50:14Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![limac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/limac/32/3981_2.png) [@limac](https://discuss.elastic.co/u/limac)\
**Post date:** [December 17, 2015, 2:50am UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425/1 "2015-12-17T02:50:14Z")

</div>

Dear all,  
I have a cluster of 6 nodes of heavy indexing works, it's 4000docs/s for each. all of those nodes have the same hardware configuration and elasticsearch configuration.  
this is of the normal node:  
 ![](https://us1.discourse-cdn.com/elastic/original/2X/8/8264de316a82677bce44be72dbbb1d191926013a.png)  
and this is the node with much higher write operations:  
 ![](https://us1.discourse-cdn.com/elastic/original/2X/1/172ae31286a4e64ae95a7f15f8e38ceaf2ac5d4c.png)  
why there is 50 times difference of these 2 nodes?

the elasticsearch version is 1.3.0 and doc value is enabled.  
thanks.

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [December 17, 2015, 9:10am UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425/2 "2015-12-17T09:10:58Z")

</div>

Maybe it's the master node?

---

<div class="post-metadata">

**Author:** ![jimczi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jimczi/32/47985_2.png) [@jimczi](https://discuss.elastic.co/u/jimczi)\
**Post date:** [December 17, 2015, 9:37am UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425/3 "2015-12-17T09:37:18Z")

</div>

Can you share a bit about your deployment (how many replicas/shards...) ? Are you using any routing for your documents ?

---

<div class="post-metadata">

**Author:** ![limac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/limac/32/3981_2.png) [@limac](https://discuss.elastic.co/u/limac)\
**Post date:** [December 17, 2015, 11:43am UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425/4 "2015-12-17T11:43:30Z")

</div>

this node is not master, and we have daily index with 6 shards+0replica and each shard a node.  
And I just found that though this node has same number of file write operation with current master node, and it is 50 times then other nodes. but the write bytes of all nodes are same!

---

<div class="post-metadata">

**Author:** ![limac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/limac/32/3981_2.png) [@limac](https://discuss.elastic.co/u/limac)\
**Post date:** [December 29, 2015, 4:10am UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425/5 "2015-12-29T04:10:59Z")

</div>

after a few study on this, I found this node had same write size with other nodes. So, I ran iostat on these nodes and found this node had much higher wrqm/s and very small avgrq-sz , and that's why bigdesk showed there are so many write operations and same write size. but why? any comment?

thanks!

---

<div class="post-metadata">

**Author:** ![jimczi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jimczi/32/47985_2.png) [@jimczi](https://discuss.elastic.co/u/jimczi)\
**Post date:** [December 29, 2015, 6:20pm UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425/6 "2015-12-29T18:20:08Z")

</div>

What method do you use to send your data to the nodes, bulk indexing ? If not then you should try it otherwise can you share the hot\_threads outputs of the nodes: [https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html)

---

<div class="post-metadata">

**Author:** ![limac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/limac/32/3981_2.png) [@limac](https://discuss.elastic.co/u/limac)\
**Post date:** [December 30, 2015, 1:18am UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425/7 "2015-12-30T01:18:11Z")

</div>

I have found the difference, that is the file system of this node is ext4, all the others are xfs!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:27pm UTC](https://discuss.elastic.co/t/one-of-the-nodes-in-cluster-has-much-higher-number-of-write-operations-than-other-nodes/37425/8 "2017-07-05T23:27:57Z")

</div>


