Hey,
A year +- we decided to change our ingesting pipeline to work with coordinating nodes. (https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#coordinating-node)
This helped us a lot, all the networking work routed to these nodes and the data nodes started to index much faster.
Few weeks ago we decided to update the stack to 7.8.1 from 7.4.0, the moment we finished the update we saw a huge rise in CPU usage + load on all the data nodes and in the same time a drop on the coordinating nodes.
Our guess was that elastic changed the way coordinating nodes handle network and data routing across the cluster, we didn't found anything that supports this claim online, the only thing we saw is the rise in CPU usage and that threads are now working on network as well on data nodes.
So now be are back to the problem we had a year ago, too much load on data nodes which do not let us to ingest as fast as we want.
Is there another way to work correctly with coordinating nodes?
Anyone experienced this before?
Any suggestions?
Thanks!