# Flush.total numbers are big different in cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [January 4, 2019, 2:30am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886 "2019-01-04T02:30:19Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![jasony](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@jasony](https://discuss.elastic.co/u/jasony)\ **Post date:** [January 4, 2019, 2:30am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/1 "2019-01-04T02:30:19Z")
Hello, My cluster consists of 9nodes and uses hot-warm architecture. 01: coordinating 02,03: master eligible nodes 04,05,06: hot 07,08,09: warm I have questions about flush.total results from \_cat api's. As we see from below results, flush.total numbers were totally different from each nodes(warm). Is it possible to make them equal (or close to)? so each node work equally. I also checked with hot\_threads api - 08 node is doing nothing and only 07,09 nodes are busy for "[flush]". last batch (via curator:allocation) was run 22hours ago. Please advice if there are any setting to make them equally receiving traffics and executing flush. Thanks! * * * GET \_cluster/health { "cluster\_name": "gm", "status": "green", "timed\_out": false, "number\_of\_nodes": 9, "number\_of\_data\_nodes": 6, "active\_primary\_shards": 1215, "active\_shards": 2430, "relocating\_shards": 0, "initializing\_shards": 0, "unassigned\_shards": 0, "delayed\_unassigned\_shards": 0, "number\_of\_pending\_tasks": 0, "number\_of\_in\_flight\_fetch": 0, "task\_max\_waiting\_in\_queue\_millis": 0, "active\_shards\_percent\_as\_number": 100 } GET \_cat/nodes?v&s=id&s=name&h=name,flush.total,flush.total\_time name flush.total flush.total\_time suy-prd-opr-els-01 0 0s suy-prd-opr-els-02 0 0s suy-prd-opr-els-03 0 0s suy-prd-opr-els-04 353452399 1.9h suy-prd-opr-els-05 1217076786 1.9h suy-prd-opr-els-06 8410385606 3h suy-prd-opr-els-07 14970158534 2.7h suy-prd-opr-els-08 346690902 3.7m suy-prd-opr-els-09 66519382342 9.8h GET \_cat/nodes?v&s=name&h=name,disk.total,disk.used,disk.used\_percent name disk.total disk.used disk.used\_percent suy-prd-opr-els-01 47.6gb 5.7gb 11.97 suy-prd-opr-els-02 47.6gb 5.3gb 11.17 suy-prd-opr-els-03 47.6gb 5.3gb 11.14 suy-prd-opr-els-04 499.7gb 169.7gb 33.97 suy-prd-opr-els-05 499.7gb 178.8gb 35.79 suy-prd-opr-els-06 499.7gb 173.4gb 34.70 suy-prd-opr-els-07 3.9tb 2.8tb 74.12 suy-prd-opr-els-08 3.9tb 2.6tb 68.29 suy-prd-opr-els-09 3.9tb 2.6tb 68.92 ---
**Author:** ![bloke](https://avatars.discourse-cdn.com/v4/letter/b/b5e925/32.png) [@bloke](https://discuss.elastic.co/u/bloke)\ **Post date:** [January 5, 2019, 1:32am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/2 "2019-01-05T01:32:00Z")
Can you please advise if you are using routing or shard allocation as described here [https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html) This type of configuration can assist in loading and balancing ---
**Author:** ![jasony](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@jasony](https://discuss.elastic.co/u/jasony)\ **Post date:** [January 5, 2019, 2:28am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/3 "2019-01-05T02:28:07Z")
i don't use and leave them as default, but my nodes are not in same zone nor rack. ---
**Author:** ![bloke](https://avatars.discourse-cdn.com/v4/letter/b/b5e925/32.png) [@bloke](https://discuss.elastic.co/u/bloke)\ **Post date:** [January 5, 2019, 3:07am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/4 "2019-01-05T03:07:15Z")
Perhaps that is a reason why you should consider using it 🙂 If you have a smaller development cluster you could try the results of these settings I use this for example. [https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html) in this type of capacity ``` node.name: node-1 # Add custom attributes to the node: # node.attr.dc_id: THIS_DC cluster.routing.allocation.awareness.attributes: dc_id ``` Another node ``` node.name: node-2 # # Add custom attributes to the node: # node.attr.dc_id: THAT_DC cluster.routing.allocation.awareness.attributes: dc_id ``` I have more nodes in the cluster than just 2 but you get the understanding i hope you may want to try these settings and experiment on something smaller I use it because i can spread the load and allocations for THIS\_DC or THAT\_DC as i like, you can customise you attributes to suite... I hope this helps you in some way ---
**Author:** ![jasony](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@jasony](https://discuss.elastic.co/u/jasony)\ **Post date:** [January 5, 2019, 3:41am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/5 "2019-01-05T03:41:16Z")
> [@bloke](#): > > node-1 # Add custom attributes to the node: # node.attr.dc\_id: THIS\_DC i will take a look. thanks! ---
**Author:** ![bloke](https://avatars.discourse-cdn.com/v4/letter/b/b5e925/32.png) [@bloke](https://discuss.elastic.co/u/bloke)\ **Post date:** [January 5, 2019, 4:02am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/6 "2019-01-05T04:02:58Z")
What type of disk are you writing to? How many replicas are you dealing with in your indexes? I have the following stats - but i have flash array and not tiered like you ``` name flush.total flush.total_time lb-kibana 0 0s node-1 4581 11m node-2 3357 1.2m node-3 4506 36.2m node-4 4470 33.7m node-5 2922 2.8m ``` And ``` name disk.total disk.used disk.used_percent lb-kibana 3.5gb 1.9gb 56.53 node-1 2.9tb 1.1tb 39.42 node-2 2.8tb 505.8gb 17.13 node-3 2.8tb 1.3tb 45.13 node-4 2.8tb 1.3tb 45.68 node-5 2.8tb 604.3gb 20.47 ``` nodes 1 3 and 4 are in a group and nodes 2 and 5 are in another the lb-kibana node is a load balancer for kibana Your flush times are higher due to something like shard allocation, replicas, infrastructure etc.. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 5, 2019, 5:39am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/7 "2019-01-05T05:39:21Z")
The stats are counters initiated at startup as far as I recall and the node with lower value seems to have a much shorter uptime than the others which could explain the difference? ---
**Author:** ![jasony](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@jasony](https://discuss.elastic.co/u/jasony)\ **Post date:** [January 7, 2019, 1:46am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/8 "2019-01-07T01:46:25Z")
I queried again for each node's uptime info, but they were up at the same time. **GET \_cat/nodes?v&s=id&s=name&h=name,flush.total,flush.total\_time,uptime** name flush.total flush.total\_time uptime suy-prd-opr-els-01 0 0s 64.8d suy-prd-opr-els-02 0 0s 64.7d suy-prd-opr-els-03 0 0s 64.7d suy-prd-opr-els-04 353463374 2h 71.6d suy-prd-opr-els-05 1217078363 1.9h 71.6d suy-prd-opr-els-06 8410387267 3.1h 71.6d suy-prd-opr-els-07 236892469127 1.1d 4.8d suy-prd-opr-els-08 78613201854 11.2h 4.8d suy-prd-opr-els-09 304126539811 1.7d 4.8d ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 7, 2019, 6:13am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/9 "2019-01-07T06:13:25Z")
It look like I did indeed misread the data. Then I am not sure what is going on. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 4, 2019, 6:13am UTC](https://discuss.elastic.co/t/flush-total-numbers-are-big-different-in-cluster/162886/10 "2019-02-04T06:13:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.