I have current logging delivery flow:
Filebeat -> Redis -> Logstash -> Elasticsearch
Redis and Logstash are deployed in the same server. So I have array of (Redis + Logstash) servers and I whant to load balance my filebeat output requests to this servers array. Documentation says that I need to provide an array of IP's or hostnames and set loadbalance to true in redis config section on filebeat side. When I did it I got traffic jump on all my servers from 10 Mbit/s to 700 Mbit/s. But if I set loadbalance option to false (as Documentation says that server was chosen randomly from array) the traffic becomes normal to 10 Mbit/s. In both cases amout of logs were not changed. I have no clue what happens.
My filebeat configuration file:
filebeat.prospectors: - input_type: log paths: - "/var/log/containers/*.log" document_type: container-logs symlinks: true tail_files: true output.redis: hosts: ["10.0.0.1","10.0.0.2","10.0.0.3"] password: "some password" datatype: "list" key: "filebeat" db: 0 loadbalance: true worker: 1 logging: level: error