One of the nodes in cluster has much higher number of write operations than other nodes

Elastic Stack Elasticsearch
1

Dear all,
I have a cluster of 6 nodes of heavy indexing works, it's 4000docs/s for each. all of those nodes have the same hardware configuration and elasticsearch configuration.
this is of the normal node:

and this is the node with much higher write operations:

why there is 50 times difference of these 2 nodes?

the elasticsearch version is 1.3.0 and doc value is enabled.
thanks.

2

Maybe it's the master node?

3

Can you share a bit about your deployment (how many replicas/shards...) ? Are you using any routing for your documents ?

4

this node is not master, and we have daily index with 6 shards+0replica and each shard a node.
And I just found that though this node has same number of file write operation with current master node, and it is 50 times then other nodes. but the write bytes of all nodes are same!

5

after a few study on this, I found this node had same write size with other nodes. So, I ran iostat on these nodes and found this node had much higher wrqm/s and very small avgrq-sz , and that's why bigdesk showed there are so many write operations and same write size. but why? any comment?

thanks!

6

What method do you use to send your data to the nodes, bulk indexing ? If not then you should try it otherwise can you share the hot_threads outputs of the nodes: https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-hot-threads.html

7

I have found the difference, that is the file system of this node is ext4, all the others are xfs!