Too much network data out in 8.4.3 elasticsearch

Elastic Stack Elasticsearch
1

HI

We have migrated from elasticsearch 6.2.3 to 8.4.3 and seeing huge network data transfer cost.is anyone else also facing this issue or its suppose to be happen.

in my configuration only 3 node cluster all are mater eligible and data nodes.Most of the network out is happening on es-1 which current master.

earlier we have data out cost 70$ month now we have 400$ for 20days which am expecting something is wrong with conf.
sharing parameter below if anyone can help on this matter:-

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

Elasticsearch Guide | Elastic

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

cluster.name: hk-app

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

node.name: es-node-1

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

By default Elasticsearch is only accessible on localhost. Set a different

address here to expose this node on the network:

network.host: 0.0.0.0

By default Elasticsearch listens for HTTP traffic on the first free port it

finds starting at 9200. Set a specific HTTP port here:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

discovery.seed_hosts: ["10.0.60.93", "10.0.70.93", "10.0.80.93"]

--------------------------------- Readiness ----------------------------------

Enable an unauthenticated TCP readiness endpoint on localhost

#readiness.port: 9399

---------------------------------- Various -----------------------------------

Allow wildcard deletion of indices:

#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------

The following settings, TLS certificates, and keys have been automatically

generated to configure Elasticsearch security features on 25-01-2023 06:15:42

--------------------------------------------------------------------------------

Enable security features

xpack.security.enabled: true

xpack.security.enrollment.enabled: false

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12

Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:
enabled: true
verification_mode: full
keystore.path: certs/es-node-1.p12
truststore.path: certs/es-node-1.p12
logger.org.elasticsearch.xpack.core.ssl: TRACE

Create a new cluster with the current node only

Additional nodes can still join the cluster later

cluster.initial_master_nodes: ["10.0.60.93", "10.0.70.93", "10.0.90.93"]

Allow HTTP API connections from anywhere

Connections are encrypted and require user authentication

#http.host: 0.0.0.0

Allow other nodes to join the cluster from anywhere

Connections are encrypted and mutually authenticated

#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

2

elasticsearch 6.2.3 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

We know that's why we upgraded.

4

Can you format your config using the Preformatted Text? The </> button? It is pretty hard to read the way it is.

Also, can you provide more context on how your cluster is deployed? What is this network data cost? Are your nodes in different datacenters?

Do you have anything in the logs that would indicate an issue?

5

  1. i have removed some comments and keep point out things.Please check now.

  2. Its 3 node cluster each node is deployed in different availability zone in the same region(Mumbai).These nodes are running on aws cloud.

  3. Network data out cost is something called in aws billing terms is data transfer or data out

  4. In the logs i have only found related to Geoip processor and downloading .mmdb file
    <
    [2023-01-25T11:47:59,638][INFO ][o.e.i.g.GeoIpDownloader ] [es-node-1] successfully downloaded geoip database [GeoLite2-ASN.mmdb]
    146 [2023-01-25T11:47:59,940][INFO ][o.e.i.g.DatabaseNodeService] [es-node-1] successfully loaded geoip database file [GeoLite2-ASN.mmdb]
    147 [2023-01-25T11:48:02,502][INFO ][o.e.i.g.GeoIpDownloader ] [es-node-1] successfully downloaded geoip database [GeoLite2-City.mmdb]
    148 [2023-01-25T11:48:03,652][INFO ][o.e.i.g.DatabaseNodeService] [es-node-1] successfully loaded geoip database file [GeoLite2-City.mmdb]
    149 [2023-01-25T11:48:03,813][INFO ][o.e.i.g.GeoIpDownloader ] [es-node-1] successfully downloaded geoip database [GeoLite2-Country.mmdb]
    150 [2023-01-25T11:48:03,893][INFO ][o.e.i.g.DatabaseNodeService] [es-node-1] successfully lo
    />

  5. Seems to me all traffic ending on master nodes.Its suppose to be doing load balancing internally.

Config FILE:-

Use a descriptive name for your cluster:

< cluster.name: hk-app />

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

< node.name: es-node-1 />

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

< path.data: /var/lib/elasticsearch />

Path to log files:

< path.logs: /var/log/elasticsearch />

---------------------------------- Network -----------------------------------

Set the host and port number for network access:

<
network.host: 0.0.0.0
http.port: 9200
/>

Set the seed hosts for cluster discovery:

discovery.seed_hosts: ["10.0.60.93", "10.0.70.93", "10.0.80.93"]

Enable security features

<
xpack.security.enabled: true
xpack.security.enrollment.enabled: false
/>

Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents

<
xpack.security.http.ssl:
enabled: false
keystore.path: certs/http.p12
/>

Enable encryption and mutual authentication between cluster nodes

<
xpack.security.transport.ssl:
enabled: true
verification_mode: full
keystore.path: certs/es-node-1.p12
truststore.path: certs/es-node-1.p12
logger.org.elasticsearch.xpack.core.ssl: TRACE
/>

Create a new cluster with the current node only

Additional nodes can still join the cluster later

<
cluster.initial_master_nodes: ["10.0.60.93", "10.0.70.93", "10.0.90.93"]
/>

Allow other nodes to join the cluster from anywhere

Connections are encrypted and mutually authenticated

#transport.host: 0.0.0.0

6

If possible edit your post and use the preformatted text button as asked to make your configuration more readable, it is the </> button.

But I don't think there is any issue in your configuration, there is nothing in the configuration that you could change to reduce the data between nodes.

You made a big jump from 6.2 to 8.4 and a lot has change in Elasticsearch.

How did you upgrade? Because you can't directly upgrade from 6.2 to 8.4, did you upgrade to 7.17 first? Did you create a new cluster?

Is this still happening? Because maybe it was just an initial rebalance after the upgrade, this would probably move a lot of shards between nodes and since AWS charges for ec2 intra-zone data transfer, this would increase your bill.

7
  1. we have created new fresh elastic cluster on 8.4.3
  2. Yes it's still happening
  3. Can you help me with loadbalancing of elasticsearch in earlier version its happening automatically now all the requests ending on current master

How can i balance my requests do i need to put load balancer in front of them.

8

Not sure what you mean with that, what is making request to Elasticsearch? Can you provide more context?

9

in previous version traffic is distributed on all nodes of es cluster but in new es cluster
all requests has been ended on current master so thats why am asking for to balance load to all node do i have manage load balancing externally in new versions or how its happen.

10

Load balancing is done by the client(s) sending data to Elasticsearch. What are you using to index data? How is this configured?

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.