# Load balancing with a single Logstash output **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [July 18, 2016, 12:01pm UTC](https://discuss.elastic.co/t/load-balancing-with-a-single-logstash-output/55756 "2016-07-18T12:01:19Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![joshuaspence](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshuaspence/32/10939_2.png) [@joshuaspence](https://discuss.elastic.co/u/joshuaspence)\ **Post date:** [July 18, 2016, 12:01pm UTC](https://discuss.elastic.co/t/load-balancing-with-a-single-logstash-output/55756/1 "2016-07-18T12:01:19Z")
I am trying to use Filebeat and have the output (Logstash) hosts queried via [Consul's DNS interface](https://www.consul.io/docs/agent/dns.html). My Filebeat configuration looks like this: ``` --- filebeat: spool_size: 2048 idle_timeout: '10s' registry_file: '.filebeat' publish_async: true config_dir: /etc/filebeat/conf.d output: logstash: hosts: - 'logstash.service.consul:5044' loadbalance: true ``` The `/etc/filebeat/conf.d` directory contains a single prospector: ``` --- filebeat: prospectors: - paths: - '/mnt/logs/access.log' encoding: 'plain' fields_under_root: false input_type: 'log' document_type: 'nginx-access' partial_line_waiting: '5s' ``` If I query Consul's DNS interface directly, I get multiple results (as expected): ``` > dig +short logstash.service.consul 10.136.110.235 10.31.92.114 10.149.152.205 ``` It doesn't seem that Filebeats understands, however, that `logstash.service.consul` points to multiple Logstash hosts. Running `filebeat -httpprof 127.0.0.1:6060` and querying [http://127.0.0.1:6060/debug/vars](http://127.0.0.1:6060/debug/vars), I get a measly ~150 events/second (using the `expvar_rates.py` script). Making the following change to the configuration file, I am able to process ~500 events/second (which still isn't great): ``` --- output: logstash: hosts: - '10.136.110.235:5044' - '10.31.92.114:5044' - '10.149.152.205:5044' loadbalance: true ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 18, 2016, 5:03pm UTC](https://discuss.elastic.co/t/load-balancing-with-a-single-logstash-output/55756/2 "2016-07-18T17:03:12Z")
which filebeat and logstash version have you installed. Using most recent 5.0 alpha versions of filebeat and logstash, you can try to enable [network pipelining](https://www.elastic.co/guide/en/beats/filebeat/master/logstash-output.html#_pipelining) mode, in order to decrease network latencies. beats connecting to logstash, query for all known IPs and choose one IP by chance. For better load-balancing support you can increase `output.logstash.worker` or configure all known hosts (or some hosts multiple times). Every host configured gets a total of `output.logstash.worker` workers assigned. That is, your first config will get you 1 output worker and the second with all hosts configured gets you 3 output workers. If filebeat gets back-pressure from output, it will slow down generating events (reading files). Without knowing any details about hardware, logstash or filters in logstash, number of beats connecting to one logstash instance, support elasticsearch ingestion rate (baseline performance in general), I have a hard time commenting on number of events/second. There is currently some [effort re-implementing beats input plugin in java](https://github.com/logstash-plugins/logstash-input-beats/issues/92), which should help a lot with performance (given logstash filters or output is not the bottleneck). I think [3.1.0-beta1](https://rubygems.org/gems/logstash-input-beats/versions/3.1.0.beta1-java) is based on java-based code base, but I wouldn't use it for production yet. ---
**Author:** ![joshuaspence](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshuaspence/32/10939_2.png) [@joshuaspence](https://discuss.elastic.co/u/joshuaspence)\ **Post date:** [July 20, 2016, 12:51pm UTC](https://discuss.elastic.co/t/load-balancing-with-a-single-logstash-output/55756/3 "2016-07-20T12:51:31Z")
I'm not sure that I understand your answer. Essentially, I'm wondering how/when Filebeat resolves DNS and how/if Filebeat handles DNS records that resolve to multiple IP addresses. Specifically, if I point Filebeat to a Logstash host with a DNS name of `logstash.service.consul`, which resolves to three IP addresses, can I expect Filebeat to load balance traffic across these three IP addresses? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 20, 2016, 1:59pm UTC](https://discuss.elastic.co/t/load-balancing-with-a-single-logstash-output/55756/4 "2016-07-20T13:59:32Z")
> if I point Filebeat to a Logstash host with a DNS name of logstash.service.consul, which resolves to three IP addresses, can I expect Filebeat to load balance traffic across these three IP addresses? No. > I'm wondering how/when Filebeat resolves DNS Filebeat queries for a list of all known IPs for some given host name. > how/if Filebeat handles DNS records that resolve to multiple IP addresses Filebeat selects one of the IP-addresses by random and attempts to create exactly one network connection. You want to enable load-balancing either add all known IPs or set `output.logstash.worker: 3` in filebeat. ---
**Author:** ![joshuaspence](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshuaspence/32/10939_2.png) [@joshuaspence](https://discuss.elastic.co/u/joshuaspence)\ **Post date:** [July 20, 2016, 9:02pm UTC](https://discuss.elastic.co/t/load-balancing-with-a-single-logstash-output/55756/5 "2016-07-20T21:02:25Z")
> [@steffens](#): > > You want to enable load-balancing either add all known IPs or set output.logstash.worker: 3 in filebeat. So if I set `output.logstash.worker: 3` then Filebeat will randomly resolve `logstash.service.consul` to an IP address three times? Do I also need to enable the `loadbalance` setting? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [July 21, 2016, 2:51pm UTC](https://discuss.elastic.co/t/load-balancing-with-a-single-logstash-output/55756/6 "2016-07-21T14:51:59Z")
> So if I set output.logstash.worker: 3 then Filebeat will randomly resolve logstash.service.consul to an IP address three times? yes > Do I also need to enable the loadbalance setting? yes ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 8, 2016, 12:01pm UTC](https://discuss.elastic.co/t/load-balancing-with-a-single-logstash-output/55756/7 "2016-08-08T12:01:27Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.