We have been given a logstash endpoint, which goes against 2 replicas running on an infra of k8s. What we see is that there is no control of the logstash pod to which we connect, and that many times our clients end up in the same pod decreasing the throughput by half, since we see that the sessions that open the filebeats are constantly open until they have nothing to send.
Does the filebeat have any configuration options for the connection against the logstash to close and open every X seconds, every X events, or every X bytes?
I have also tried to increase the number of connections from each client, adding in the output of logstash "worker: 2" but I see that it does not work either.
As you said, i had to set "pipelining: 0", but also "loadbalance: true". The guide reference says loadbalance is true by default, but it seems that is not true, i have had to set "loadbalance: true" in the config to work. But with this config:
With one host as i have, with only pipelining: 0, it didn't work. Then, adding "loadbalance: true", it started to work as expected.
Finally I have not been able to connect to a specific pod, I understand because my service is not headless. But I managed to control the session balancing with this configuration in the k8s service:
With this configuration you can control from the loadbalancer the sessions to each worker/pod. Specifically I have configured a least_connections type balancing so that all pods have the same number of sessions.
Thanks for your help, because with these configurations I can say that I have managed to have a balanced flow.
It seems like you're facing connection issues with Logstash and are looking for ways to optimize Filebeat. To address the connection problem, you may want to consider adjusting the loadbalance option in your Filebeat configuration. Additionally, you can explore using the timeout setting to manage connections more effectively. Keep in mind that tuning these parameters may require some experimentation to find the best configuration for your specific use case. Best of luck with your optimization efforts! AC Football Cases