Can I add loadbalance to filebeat config for hosts in logstash output?

Elastic Stack Beats
1

https://www.elastic.co/guide/en/beats/filebeat/current/logstash-output.html

It says I can add a list of hosts in logstash output and enable loadbalancer

but I am not sure if I can just add the loadbalancer like : logstashhosts:5044
logstashhosts is the name of a lb for logstash

2

I don't understand. Can you post sample configs?

e.g.

filebeat:
  ...

  spool_size: 4096

...

output:
  logstash:
    hosts:
      - logstash1:5044
      - logstash2:5044
    loadbalance: true
    bulk_max_size: 2048
    worker: 1

will get you some lock-step load-balancing. For lock-step loadbalancing use spool_size = worker * #hosts * bulk_max_size. Having just one host one can do (2 load balance workers):

filebeat:
  ...

  spool_size: 4096

...

output:
  logstash:
    hosts:
      - logstash:5044
    loadbalance: true
    bulk_max_size: 2048
    worker: 2
3

sorry , let me make my question more clear.

output:
logstash:
hosts:
- logstash1:5044
- logstash2:5044

for the above configuration, can I give a loadbalancer name instead of giving a list of logstash hosts???
Or I am misunderstanding loadbalancer here?

4

what kind of load-balancer are you using?

5

ELB

you are giving a list of logstash hosts, what if the two hosts are behind a loadbalancer?

6

The AWS ELB? Logstash output is not using HTTP, but sits right on top of TCP. ELB will balance connections round-robin. That is you can use logstash with worker: 2 settings and the balancer it's address. But filebeat will keep connection until one logstash instance goes away. Starting multiple beats at the same time, there is a chance a beat might get all workers assigned to same host by ELB.

7

This topic was automatically closed after 21 days. New replies are no longer allowed.