Currently I have a ECK Cluster running on a AKS cluster. The configuration is 3 master nodes, 5 hot data nodes that also are ingest nodes and 7 warm/cold nodes. We are having performance issues (filebeat is showing write lateness of up to 40s and from time to time we have saturation queue on 1). While reading the documentation for ECk I saw that could be possible to have ingest only nodes.
My filebeat has a default pipeline configured so every event sent to ES will be processed by a pipeline.
Currently all nodes are configured to receive request from the loadbalancer service.
My questions:
- If I add ingest nodes and only those are configured to receive requests from Load Balancer it, will the node ingest coordinate the request to data node first than data node will send back to ingest node in order to apply the pipeline, or it will first run the pipeline before sending to proper data node to write in the shard?
- Should I remove the master nodes to receive requests, to avoid this extra hop?
Also if you know the place in the documentation where I find this request data flow explanations could you share, I failed to find the precise place that talks about this data flows and DeepSeek is giving some strange responses regarding this.
Thanks in Advance.