# Why Elasticsearch send more data between the cluster nodes

**URL:** <https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046>\
**Category:** Elasticsearch\
**Created:** [September 24, 2021, 4:01am UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046 "2021-09-24T04:01:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![ravindrababua](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@ravindrababua](https://discuss.elastic.co/u/ravindrababua)\
**Post date:** [September 24, 2021, 4:01am UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046/1 "2021-09-24T04:01:41Z")

</div>

We have observed ELK sending the data with in the cluster nodes, we are not sending large data as a input to Elasticsearch.

when the data is tool large it throws circuit break exception and cluster goes to YELLOW, but immediately with in few seconds it comes to the normal state.

why it will send ( or replicate ) large data with in the cluster node at a time ?

once data transmission is completed it comes to the normal state.

```auto
[2021-08-09T01:11:30,001][WARN][o.e.a.b.TransportShardBulkAction] [rep-s02] [[rstraffic-2021.08.08][0]] failed to perform indices:data/write/bulk[s] on replica [rstraffic-2021.08.08][0], node[v2-ht7q_RcuEQWt-P0yJmg], [R], s[STARTED], a[id=eUpGCnydT3iIMjm76Qg3zQ]
org.elasticsearch.transport.RemoteTransportException: [rep-s03]

```

```auto
[192.25.41.12:9300][indices:data/write/bulk[s][r]]
Caused by: org.elasticsearch.common.breaker.CircuitBreakingException: [parent] Data too large, data for [<transport_request>] would be [20465471546/19gb], which is larger than the limit of [20401094656/19gb], real usage: [20465468584/19gb], new bytes reserved: [2962/2.8kb], usages [request=0/0b, fielddata=11621082/11mb, in_flight_requests=9038/8.8kb, accounting=55508596/52.9mb]

```

```auto
[2021-08-09T01:11:30,013][INFO][o.e.c.r.a.AllocationService] [rep-s02] Cluster health status changed from [GREEN] to [YELLOW] (reason: [shards failed [[rstraffic-2021.08.08][0]]]).
[2021-08-09T01:11:32,895][INFO][o.e.c.r.a.AllocationService] [rep-s02] Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards started [[rstraffic-2021.08.08][0]]]).

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 24, 2021, 5:10am UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046/2 "2021-09-24T05:10:30Z")

</div>

What is the output from the `_cluster/stats?pretty&human` API?

---

<div class="post-metadata">

**Author:** ![ravindrababua](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@ravindrababua](https://discuss.elastic.co/u/ravindrababua)\
**Post date:** [September 24, 2021, 12:50pm UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046/3 "2021-09-24T12:50:10Z")

</div>

> [@warkolm](#):
>
> \_cluster/stats?pretty&human

Please find my output below for the above API

{  
"\_nodes" : {  
"total" : 3,  
"successful" : 3,  
"failed" : 0  
},  
"cluster\_name" : "mainelk",  
"cluster\_uuid" : "6yfpVUZYRVWFdeHWYU6tqg",  
"timestamp" : 1632487383878,  
"status" : "green",  
"indices" : {  
"count" : 235,  
"shards" : {  
"total" : 470,  
"primaries" : 235,  
"replication" : 1.0,  
"index" : {  
"shards" : {  
"min" : 2,  
"max" : 2,  
"avg" : 2.0  
},  
"primaries" : {  
"min" : 1,  
"max" : 1,  
"avg" : 1.0  
},  
"replication" : {  
"min" : 1.0,  
"max" : 1.0,  
"avg" : 1.0  
}  
}  
},  
"docs" : {  
"count" : 187619472,  
"deleted" : 10941  
},  
"store" : {  
"size" : "162.9gb",  
"size\_in\_bytes" : 174951161873  
},  
"fielddata" : {  
"memory\_size" : "177.3kb",  
"memory\_size\_in\_bytes" : 181584,  
"evictions" : 0  
},  
"query\_cache" : {  
"memory\_size" : "1.8mb",  
"memory\_size\_in\_bytes" : 1979421,  
"total\_count" : 276530139,  
"hit\_count" : 252442544,  
"miss\_count" : 24087595,  
"cache\_size" : 284,  
"cache\_count" : 579947,  
"evictions" : 579663  
},  
"completion" : {  
"size" : "0b",  
"size\_in\_bytes" : 0  
},  
"segments" : {  
"count" : 4830,  
"memory" : "166.8mb",  
"memory\_in\_bytes" : 174921722,  
"terms\_memory" : "139.9mb",  
"terms\_memory\_in\_bytes" : 146741392,  
"stored\_fields\_memory" : "4.1mb",  
"stored\_fields\_memory\_in\_bytes" : 4370304,  
"term\_vectors\_memory" : "0b",  
"term\_vectors\_memory\_in\_bytes" : 0,  
"norms\_memory" : "18.2mb",  
"norms\_memory\_in\_bytes" : 19123584,  
"points\_memory" : "0b",  
"points\_memory\_in\_bytes" : 0,  
"doc\_values\_memory" : "4.4mb",  
"doc\_values\_memory\_in\_bytes" : 4686442,  
"index\_writer\_memory" : "1.8gb",  
"index\_writer\_memory\_in\_bytes" : 2006032040,  
"version\_map\_memory" : "0b",  
"version\_map\_memory\_in\_bytes" : 0,  
"fixed\_bit\_set" : "107.8kb",  
"fixed\_bit\_set\_memory\_in\_bytes" : 110464,  
"max\_unsafe\_auto\_id\_timestamp" : 1632441621559,  
"file\_sizes" : { }  
},  
"mappings" : {  
"field\_types" : [  
{  
"name" : "alias",  
"count" : 374,  
"index\_count" : 11  
},

```
    {
      "name" : "text",
      "count" : 14031,
      "index_count" : 233
    }
  ]
},
"analysis" : {
  "char_filter_types" : [],
  "tokenizer_types" : [],
  "filter_types" : [],
  "analyzer_types" : [],
  "built_in_char_filters" : [],
  "built_in_tokenizers" : [],
  "built_in_filters" : [],
  "built_in_analyzers" : []
}

```

},  
"nodes" : {  
"count" : {  
"total" : 3,  
"coordinating\_only" : 0,  
"data" : 3,  
"ingest" : 0,  
"master" : 3,  
"ml" : 0,  
"remote\_cluster\_client" : 0,  
"transform" : 3,  
"voting\_only" : 0  
},  
"versions" : [  
"7.7.0"  
],  
"os" : {  
"available\_processors" : 36,  
"allocated\_processors" : 36,  
"names" : [  
{  
"name" : "Linux",  
"count" : 3  
}  
],  
"pretty\_names" : [  
{  
"pretty\_name" : "Red Hat Enterprise Linux",  
"count" : 3  
}  
],  
"mem" : {  
"total" : "188.2gb",  
"total\_in\_bytes" : 202160398336,  
"free" : "3.7gb",  
"free\_in\_bytes" : 4024860672,  
"used" : "184.5gb",  
"used\_in\_bytes" : 198135537664,  
"free\_percent" : 2,  
"used\_percent" : 98  
}  
},  
"process" : {  
"cpu" : {  
"percent" : 1  
},  
"open\_file\_descriptors" : {  
"min" : 1779,  
"max" : 1840,  
"avg" : 1817  
}  
},  
"jvm" : {  
"max\_uptime" : "120.9d",  
"max\_uptime\_in\_millis" : 10447587500,  
"versions" : [  
{  
"version" : "14",  
"vm\_name" : "OpenJDK 64-Bit Server VM",  
"vm\_version" : "14+36",  
"vm\_vendor" : "AdoptOpenJDK",  
"bundled\_jdk" : true,  
"using\_bundled\_jdk" : true,  
"count" : 3  
}  
],  
"mem" : {  
"heap\_used" : "21.4gb",  
"heap\_used\_in\_bytes" : 23056379088,  
"heap\_max" : "60gb",  
"heap\_max\_in\_bytes" : 64424509440  
},  
"threads" : 595  
},  
"fs" : {  
"total" : "1.1tb",  
"total\_in\_bytes" : 1287860256768,  
"free" : "1tb",  
"free\_in\_bytes" : 1109414490112,  
"available" : "1tb",  
"available\_in\_bytes" : 1109414490112  
},  
"plugins" : ,  
"network\_types" : {  
"transport\_types" : {  
"security4" : 3  
},  
"http\_types" : {  
"security4" : 3  
}  
},  
"discovery\_types" : {  
"zen" : 3  
},  
"packaging\_types" : [  
{  
"flavor" : "default",  
"type" : "rpm",  
"count" : 3  
}  
],  
"ingest" : {  
"number\_of\_pipelines" : 2,  
"processor\_stats" : {  
"gsub" : {  
"count" : 0,  
"failed" : 0,  
"current" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0  
},  
"script" : {  
"count" : 0,  
"failed" : 0,  
"current" : 0,  
"time" : "0s",  
"time\_in\_millis" : 0  
}  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![ravindrababua](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@ravindrababua](https://discuss.elastic.co/u/ravindrababua)\
**Post date:** [September 27, 2021, 5:33am UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046/5 "2021-09-27T05:33:07Z")

</div>

Could you please provide update on the issue.

---

<div class="post-metadata">

**Author:** ![ravindrababua](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@ravindrababua](https://discuss.elastic.co/u/ravindrababua)\
**Post date:** [October 4, 2021, 3:49am UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046/6 "2021-10-04T03:49:20Z")

</div>

Hi Team,  
could you please provide any update on this.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [October 5, 2021, 4:20am UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046/7 "2021-10-05T04:20:26Z")

</div>

Do you have any non-standard settings in your cluster config? What kind of load is the cluster under? Are you monitoring heap usage? If so, what does it look like?

---

<div class="post-metadata">

**Author:** ![ravindrababua](https://avatars.discourse-cdn.com/v4/letter/r/71e660/32.png) [@ravindrababua](https://discuss.elastic.co/u/ravindrababua)\
**Post date:** [October 5, 2021, 6:36am UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046/8 "2021-10-05T06:36:01Z")

</div>

We have configured 3 node cluster (2master and 1data node),  
Logstash will send the data to all the Elasticsearch nodes. (In logstash we have configured all the Elasticsearch nodes)  
What ever the data comes into the Elasticsearch node will be replicated to the other cluster members.  
We have configured max/min heap is 20GB

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/e/ee24915916622586b167afb359b6a499109146c8.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/7/77f1472dffd777c170324624b80a34a19e4d1b8e.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8c028c342e95d35bd2fb2d17a5730b6031b38be0.png)  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/6/866fa16b58546ae221dd957040af4791e836075f.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 2, 2021, 6:36am UTC](https://discuss.elastic.co/t/why-elasticsearch-send-more-data-between-the-cluster-nodes/285046/9 "2021-11-02T06:36:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.