Hi Everyone!
We are trying to migrate from the transport client to the high level rest client in our application, but we noticed a strange network load on the master node.
Here's our setup:
ElasticSearch 6.8 OSS (no x-pack)
3 Master Nodes
49 Data nodes
2 different Indexes with 8 shards and 5 replicas
We have a few indexing jobs that run every few minutes, before the migration, using the transport client, the network traffic on master nodes is pretty a flat line, with small changes between 2 and 3 Mbps... using the Rest client, during the indexing, the network traffic on master nodes spikes up to 60 Mbps depending on the job.
If I understand correctly, master nodes should not be involved directly in indexing operations, and we verified that we use the SKIP_DEDICATED_MASTERS NodeSelectors, and these ARE dedicated masters (no ingest or other roles).
The traffic on the DATA nodes, changed from an average of 10 Mbps to an average of 25Mbps during indexing, which I guess can be explained with the change from the binary protocol to the rest json one, and the curves have not changed migrating from one type of client to the other.
The client starts with the list of master nodes, uses a sniffer to load the list of data nodes, and then uses the NodeSelector to only cycle in round robin between the data nodes.
We are also using the http.cname_in_publish_address=true to avoid the warning of deprecation
Any idea of what could cause this behavior?
