Load balancing with a single Logstash output

Elastic Stack Beats
1

I am trying to use Filebeat and have the output (Logstash) hosts queried via Consul's DNS interface. My Filebeat configuration looks like this:

---
filebeat:
  spool_size: 2048
  idle_timeout: '10s'
  registry_file: '.filebeat'
  publish_async: true
  config_dir: /etc/filebeat/conf.d
output:
  logstash:
    hosts:
      - 'logstash.service.consul:5044'
    loadbalance: true

The /etc/filebeat/conf.d directory contains a single prospector:

---
filebeat:
  prospectors:
    - paths:
      - '/mnt/logs/access.log'
      encoding: 'plain'
      fields_under_root: false
      input_type: 'log'
      document_type: 'nginx-access'
      partial_line_waiting: '5s'

If I query Consul's DNS interface directly, I get multiple results (as expected):

> dig +short logstash.service.consul
10.136.110.235
10.31.92.114
10.149.152.205

It doesn't seem that Filebeats understands, however, that logstash.service.consul points to multiple Logstash hosts. Running filebeat -httpprof 127.0.0.1:6060 and querying http://127.0.0.1:6060/debug/vars, I get a measly ~150 events/second (using the expvar_rates.py script). Making the following change to the configuration file, I am able to process ~500 events/second (which still isn't great):

---
output:
  logstash:
    hosts:
      - '10.136.110.235:5044'
      - '10.31.92.114:5044'
      - '10.149.152.205:5044'
    loadbalance: true
2

which filebeat and logstash version have you installed. Using most recent 5.0 alpha versions of filebeat and logstash, you can try to enable network pipelining mode, in order to decrease network latencies.

beats connecting to logstash, query for all known IPs and choose one IP by chance.

For better load-balancing support you can increase output.logstash.worker or configure all known hosts (or some hosts multiple times). Every host configured gets a total of output.logstash.worker workers assigned. That is, your first config will get you 1 output worker and the second with all hosts configured gets you 3 output workers.

If filebeat gets back-pressure from output, it will slow down generating events (reading files).

Without knowing any details about hardware, logstash or filters in logstash, number of beats connecting to one logstash instance, support elasticsearch ingestion rate (baseline performance in general), I have a hard time commenting on number of events/second. There is currently some effort re-implementing beats input plugin in java, which should help a lot with performance (given logstash filters or output is not the bottleneck). I think 3.1.0-beta1 is based on java-based code base, but I wouldn't use it for production yet.

3

I'm not sure that I understand your answer. Essentially, I'm wondering how/when Filebeat resolves DNS and how/if Filebeat handles DNS records that resolve to multiple IP addresses. Specifically, if I point Filebeat to a Logstash host with a DNS name of logstash.service.consul, which resolves to three IP addresses, can I expect Filebeat to load balance traffic across these three IP addresses?

4

if I point Filebeat to a Logstash host with a DNS name of logstash.service.consul, which resolves to three IP addresses, can I expect Filebeat to load balance traffic across these three IP addresses?

No.

I'm wondering how/when Filebeat resolves DNS

Filebeat queries for a list of all known IPs for some given host name.

how/if Filebeat handles DNS records that resolve to multiple IP addresses

Filebeat selects one of the IP-addresses by random and attempts to create exactly one network connection.

You want to enable load-balancing either add all known IPs or set output.logstash.worker: 3 in filebeat.

5

So if I set output.logstash.worker: 3 then Filebeat will randomly resolve logstash.service.consul to an IP address three times? Do I also need to enable the loadbalance setting?

6

So if I set output.logstash.worker: 3 then Filebeat will randomly resolve logstash.service.consul to an IP address three times?

yes

Do I also need to enable the loadbalance setting?

yes

7

This topic was automatically closed after 21 days. New replies are no longer allowed.