I have a filebeat agent that will be sending quite a lot of access log data and our current setup have 4 receiving logstash hosts. I would like to benefit from these to gain a maximum throughput.
When I read the «workers» setting description here, I'm a little puzzled:
The number of workers «per configured host» publishing events to Logstash. This is best used with load balancing mode enabled. Example: If you have 2 hosts and 3 workers, in total 6 workers are started (3 for each host).
The first part : number of workers per configured hosts publishing events to Logstash.
Which host does the «per configured hosts» means?
- The host running the agent?
- The hosts in the provided destination list?
Sorry for this one, maybe it's already stated clearly, but english is not my native...
So, if I have 4 target hosts and want to benefit from this, should I specify 4 workers?
To recap my understandings, to benefit from load balancing to a maximum I should have a config key performance settings like this:
filebeat: spool_size: 2048 # Default 2048 output: logstash: hosts: ["logs1.domain.com", "logs2.domain.com", "logs3.domain.com", "logs4.domain.com"] loadbalance: true worker: 4 bulk_max_size: 512 # Default 2048