[SOLVED] High Network Traffic on Master Nodes after change from Transport Client to REST Client (ElasticSearch 6.8)

[gotenks82 (Alex Pedini)]

Hi Everyone!
We are trying to migrate from the transport client to the high level rest client in our application, but we noticed a strange network load on the master node.
Here's our setup:
ElasticSearch 6.8 OSS (no x-pack)
3 Master Nodes
49 Data nodes
2 different Indexes with 8 shards and 5 replicas

We have a few indexing jobs that run every few minutes, before the migration, using the transport client, the network traffic on master nodes is pretty a flat line, with small changes between 2 and 3 Mbps... using the Rest client, during the indexing, the network traffic on master nodes spikes up to 60 Mbps depending on the job.

If I understand correctly, master nodes should not be involved directly in indexing operations, and we verified that we use the SKIP_DEDICATED_MASTERS NodeSelectors, and these ARE dedicated masters (no ingest or other roles).

The traffic on the DATA nodes, changed from an average of 10 Mbps to an average of 25Mbps during indexing, which I guess can be explained with the change from the binary protocol to the rest json one, and the curves have not changed migrating from one type of client to the other.

The client starts with the list of master nodes, uses a sniffer to load the list of data nodes, and then uses the NodeSelector to only cycle in round robin between the data nodes.
We are also using the http.cname_in_publish_address=true to avoid the warning of deprecation

Any idea of what could cause this behavior?

[gotenks82 (Alex Pedini)]

Hi Guys, sorry we found the cause of the issue
It was a misconfiguration of our client, we had a "conflict" of beans, one was actively using the sniffer to get the data nodes, the other was not, and was simply using the master nodes from configuration.
We realized after wards that only part of the code was using the right bean.

[warkolm (Mark Walkom)]

Thanks for sharing your solution