# Too much network data out in 8.4.3 elasticsearch

**URL:** <https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182>\
**Category:** Elasticsearch\
**Created:** [March 21, 2023, 3:56pm UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182 "2023-03-21T15:56:18Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dharampal\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dharampal_singh/32/118725_2.png) [@Dharampal\_Singh](https://discuss.elastic.co/u/Dharampal_Singh)\
**Post date:** [March 21, 2023, 3:56pm UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/1 "2023-03-21T15:56:18Z")

</div>

HI

We have migrated from elasticsearch 6.2.3 to 8.4.3 and seeing huge network data transfer cost.is anyone else also facing this issue or its suppose to be happen.

in my configuration only 3 node cluster all are mater eligible and data nodes.Most of the network out is happening on es-1 which current master.

earlier we have data out cost 70$ month now we have 400$ for 20days which am expecting something is wrong with conf.  
sharing parameter below if anyone can help on this matter:-

# ======================== Elasticsearch Configuration =========================

# 

# NOTE: Elasticsearch comes with reasonable defaults for most settings.

# Before you set out to tweak and tune the configuration, make sure you

# understand what are you trying to accomplish and the consequences.

# 

# The primary way of configuring a node is via this file. This template lists

# the most important settings you may want to configure for a production cluster.

# 

# Please consult the documentation for further information on configuration options:

# [Elasticsearch Guide | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/index.html)

# 

# ---------------------------------- Cluster -----------------------------------

# 

# Use a descriptive name for your cluster:

# 

cluster.name: hk-app

# 

# ------------------------------------ Node ------------------------------------

# 

# Use a descriptive name for the node:

# 

node.name: es-node-1

# 

# Add custom attributes to the node:

# 

#node.attr.rack: r1

# 

# ----------------------------------- Paths ------------------------------------

# 

# Path to directory where to store the data (separate multiple locations by comma):

# 

path.data: /var/lib/elasticsearch

# 

# Path to log files:

# 

path.logs: /var/log/elasticsearch

# 

# ----------------------------------- Memory -----------------------------------

# 

# Lock the memory on startup:

# 

#bootstrap.memory\_lock: true

# 

# Make sure that the heap size is set to about half the memory available

# on the system and that the owner of the process is allowed to use this

# limit.

# 

# Elasticsearch performs poorly when the system is swapping the memory.

# 

# ---------------------------------- Network -----------------------------------

# 

# By default Elasticsearch is only accessible on localhost. Set a different

# address here to expose this node on the network:

# 

network.host: 0.0.0.0

# 

# By default Elasticsearch listens for HTTP traffic on the first free port it

# finds starting at 9200. Set a specific HTTP port here:

# 

http.port: 9200

# 

# For more information, consult the network module documentation.

# 

# --------------------------------- Discovery ----------------------------------

# 

# Pass an initial list of hosts to perform discovery when this node is started:

# The default list of hosts is ["127.0.0.1", "[::1]"]

# 

discovery.seed\_hosts: ["10.0.60.93", "10.0.70.93", "10.0.80.93"]

# 

# --------------------------------- Readiness ----------------------------------

# 

# Enable an unauthenticated TCP readiness endpoint on localhost

# 

#readiness.port: 9399

# 

# ---------------------------------- Various -----------------------------------

# 

# Allow wildcard deletion of indices:

# 

#action.destructive\_requires\_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------

# 

# The following settings, TLS certificates, and keys have been automatically

# generated to configure Elasticsearch security features on 25-01-2023 06:15:42

# 

# --------------------------------------------------------------------------------

# Enable security features

xpack.security.enabled: true

xpack.security.enrollment.enabled: false

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:  
enabled: false  
keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:  
enabled: true  
verification\_mode: full  
keystore.path: certs/es-node-1.p12  
truststore.path: certs/es-node-1.p12  
logger.org.elasticsearch.xpack.core.ssl: TRACE

# Create a new cluster with the current node only

# Additional nodes can still join the cluster later

cluster.initial\_master\_nodes: ["10.0.60.93", "10.0.70.93", "10.0.90.93"]

# Allow HTTP API connections from anywhere

# Connections are encrypted and require user authentication

#http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere

# Connections are encrypted and mutually authenticated

#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2023, 3:56pm UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/2 "2023-03-21T15:56:18Z")

</div>

elasticsearch 6.2.3 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![Dharampal\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dharampal_singh/32/118725_2.png) [@Dharampal\_Singh](https://discuss.elastic.co/u/Dharampal_Singh)\
**Post date:** [March 21, 2023, 3:58pm UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/3 "2023-03-21T15:58:25Z")

</div>

We know that's why we upgraded.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 21, 2023, 6:20pm UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/4 "2023-03-21T18:20:15Z")

</div>

Can you format your config using the Preformatted Text? The `</>` button? It is pretty hard to read the way it is.

Also, can you provide more context on how your cluster is deployed? What is this network data cost? Are your nodes in different datacenters?

Do you have anything in the logs that would indicate an issue?

---

<div class="post-metadata">

**Author:** ![Dharampal\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dharampal_singh/32/118725_2.png) [@Dharampal\_Singh](https://discuss.elastic.co/u/Dharampal_Singh)\
**Post date:** [March 22, 2023, 4:28am UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/5 "2023-03-22T04:28:28Z")

</div>

1. i have removed some comments and keep point out things.Please check now.

2. Its 3 node cluster each node is deployed in different availability zone in the same region(Mumbai).These nodes are running on aws cloud.

3. Network data out cost is something called in aws billing terms is data transfer or data out

4. In the logs i have only found related to Geoip processor and downloading .mmdb file  
\<  
[2023-01-25T11:47:59,638][INFO][o.e.i.g.GeoIpDownloader] [es-node-1] successfully downloaded geoip database [GeoLite2-ASN.mmdb]  
146 [2023-01-25T11:47:59,940][INFO][o.e.i.g.DatabaseNodeService] [es-node-1] successfully loaded geoip database file [GeoLite2-ASN.mmdb]  
147 [2023-01-25T11:48:02,502][INFO][o.e.i.g.GeoIpDownloader] [es-node-1] successfully downloaded geoip database [GeoLite2-City.mmdb]  
148 [2023-01-25T11:48:03,652][INFO][o.e.i.g.DatabaseNodeService] [es-node-1] successfully loaded geoip database file [GeoLite2-City.mmdb]  
149 [2023-01-25T11:48:03,813][INFO][o.e.i.g.GeoIpDownloader] [es-node-1] successfully downloaded geoip database [GeoLite2-Country.mmdb]  
150 [2023-01-25T11:48:03,893][INFO][o.e.i.g.DatabaseNodeService] [es-node-1] successfully lo  
/\>

5. Seems to me all traffic ending on master nodes.Its suppose to be doing load balancing internally.

Config FILE:-

# Use a descriptive name for your cluster:

\< cluster.name: hk-app /\>

# ------------------------------------ Node ------------------------------------

# Use a descriptive name for the node:

\< node.name: es-node-1 /\>

# ----------------------------------- Paths ------------------------------------

# Path to directory where to store the data (separate multiple locations by comma):

\< path.data: /var/lib/elasticsearch /\>

# Path to log files:

\< path.logs: /var/log/elasticsearch /\>

# ---------------------------------- Network -----------------------------------

# Set the host and port number for network access:

\<  
network.host: 0.0.0.0  
http.port: 9200  
/\>

# Set the seed hosts for cluster discovery:

discovery.seed\_hosts: ["10.0.60.93", "10.0.70.93", "10.0.80.93"]

# Enable security features

\<  
xpack.security.enabled: true  
xpack.security.enrollment.enabled: false  
/\>

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

\<  
xpack.security.http.ssl:  
enabled: false  
keystore.path: certs/http.p12  
/\>

# Enable encryption and mutual authentication between cluster nodes

\<  
xpack.security.transport.ssl:  
enabled: true  
verification\_mode: full  
keystore.path: certs/es-node-1.p12  
truststore.path: certs/es-node-1.p12  
logger.org.elasticsearch.xpack.core.ssl: TRACE  
/\>

# Create a new cluster with the current node only

# Additional nodes can still join the cluster later

\<  
cluster.initial\_master\_nodes: ["10.0.60.93", "10.0.70.93", "10.0.90.93"]  
/\>

# Allow other nodes to join the cluster from anywhere

# Connections are encrypted and mutually authenticated

#transport.host: 0.0.0.0

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 22, 2023, 1:29pm UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/6 "2023-03-22T13:29:36Z")

</div>

If possible edit your post and use the preformatted text button as asked to make your configuration more readable, it is the `</>` button.

But I don't think there is any issue in your configuration, there is nothing in the configuration that you could change to reduce the data between nodes.

You made a big jump from 6.2 to 8.4 and a lot has change in Elasticsearch.

How did you upgrade? Because you can't directly upgrade from 6.2 to 8.4, did you upgrade to 7.17 first? Did you create a new cluster?

Is this still happening? Because maybe it was just an initial rebalance after the upgrade, this would probably move a lot of shards between nodes and since AWS charges for ec2 intra-zone data transfer, this would increase your bill.

---

<div class="post-metadata">

**Author:** ![Dharampal\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dharampal_singh/32/118725_2.png) [@Dharampal\_Singh](https://discuss.elastic.co/u/Dharampal_Singh)\
**Post date:** [March 22, 2023, 3:36pm UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/7 "2023-03-22T15:36:07Z")

</div>

1. we have created new fresh elastic cluster on 8.4.3
2. Yes it's still happening
3. Can you help me with loadbalancing of elasticsearch in earlier version its happening automatically now all the requests ending on current master

How can i balance my requests do i need to put load balancer in front of them.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 22, 2023, 3:57pm UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/8 "2023-03-22T15:57:26Z")

</div>

> [@Dharampal\_Singh](#):
>
> 1. Can you help me with loadbalancing of elasticsearch in earlier version its happening automatically now all the requests ending on current master

Not sure what you mean with that, what is making request to Elasticsearch? Can you provide more context?

---

<div class="post-metadata">

**Author:** ![Dharampal\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dharampal_singh/32/118725_2.png) [@Dharampal\_Singh](https://discuss.elastic.co/u/Dharampal_Singh)\
**Post date:** [March 23, 2023, 5:22am UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/9 "2023-03-23T05:22:28Z")

</div>

in previous version traffic is distributed on all nodes of es cluster but in new es cluster  
all requests has been ended on current master so thats why am asking for to balance load to all node do i have manage load balancing externally in new versions or how its happen.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 23, 2023, 5:43am UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/10 "2023-03-23T05:43:09Z")

</div>

Load balancing is done by the client(s) sending data to Elasticsearch. What are you using to index data? How is this configured?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 20, 2023, 5:44am UTC](https://discuss.elastic.co/t/too-much-network-data-out-in-8-4-3-elasticsearch/328182/11 "2023-04-20T05:44:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.