# Logstash output load balancing and workers settings

**URL:** <https://discuss.elastic.co/t/logstash-output-load-balancing-and-workers-settings/48176>\
**Category:** Beats\
**Created:** [April 22, 2016, 1:48pm UTC](https://discuss.elastic.co/t/logstash-output-load-balancing-and-workers-settings/48176 "2016-04-22T13:48:52Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [April 25, 2016, 11:11am UTC](https://discuss.elastic.co/t/logstash-output-load-balancing-and-workers-settings/48176/2 "2016-04-25T11:11:34Z")

</div>

Hi,

best advice is measure, measure, measure. [This post](https://discuss.elastic.co/t/filebeat-sending-data-to-logstash-seems-too-slow/37596/13) contains a python script + instructions how you can get throughput info right from filebeat. For testing have a log file prepared (I often use [NASA HTTP logs](http://ita.ee.lbl.gov/html/contrib/NASA-HTTP.html)) and delete registry files between runs. (optionsl) In addition use the null (or stdout with `dotted` codec and `pv` tool) output plugin in logstash to not generate any back-pressure from logstash.

The `worker: ...` config is really per host. The default value is 1. That is if `H=# of hosts` and `W=# worker`, then `H*W` workers doing output will be spawned. In your sample config it means you're spawning like 16 workers pushing data in total.

There are 2 options to try in your case:

1. set `filebeat.publish_async: true`. This will push batches as soon as batches are ready into the publisher pipeline. In this case I'd set `spool_size between [bulk_max_size, bulk_max_size * worker * (# of hosts)]`. If one logstash instance is not responding (or slowing down), filebeat continues publishing events using the other workers (due to async/pipelines publishing).

2. set `filebeat.spool_size = output.logstash.bulk_max * output.logstash.worker * len(output.logstash.hosts)`. This will split bathces into `output.logstash.worker * len(output.logstash.hosts)` batches when publishing, so every host gets it's share. Drawback is, if one logstash instance slows down it first takes some timeout to detect it's not responding and transmitting the sub-batch via another logstash instance, basically blocking output until all sub-batches have been ACKed.

My gut feeling tells me option 1 would have better throughput (despite gut feelings being often wrong), but in the end it's up to you to run experiments and measure your setup to figure some good configurations matching your requirements. And don't forget, the higher your throughput, the more resources will be required by filebeat and logstash to process your data.

---

_[View the full topic](https://discuss.elastic.co/t/logstash-output-load-balancing-and-workers-settings/48176)._