Send to Logstash localhost and remote hosts


#1

If Filebeat is configured to send to Logstash localhost only it works fine. This is with any version.
If Filebeat is configured to send to localhost and remote host then nothing is sent to localhost.
To re-create the output setting is:
hosts: ["localhost:3522", "192.168.0.50:3520"]
In this case output is sent correctly to 192.168.0.50 but localhost does not receive anything.
Running nc -lvp 3522 on localhost confirms.
Is this normal behaviour? I have tried loadbalancer:false, but no difference.


(Steffen Siering) #2

beats do not do any kind of event-routing or replication.

If loadbalance is disabled (default == false, also check your config as option loadbalancer does not exist) filebeat will only establish one connection by random (unfortunately we've found a bug in randomizer initialization, so it's not so random right now) and only if active connection fails, it will try to connect to another host. Enabling load balancing both logstash instances will see each a different subset of lines.

More advanced filtering and event routing can be done with logstash. But be carefull if you want to forward data to production and development systems at the same time. Logstash (and filebeat if it were to support event replication) do block if output is unavailable. That is having a crash (or just shutting down) development system will grind production system to a halt. A common way to solve this is using some queuing technology like redis or kafka, having filebeat/logstash push to queue and development/production system each pulling events from queues. The queuing system basically decouples production from development environment.


(system) #3