How do you merge the syslogs from network device into message queue when logstash scales out?

Elastic Stack Logstash
1

I'd like to ask question about the theory of scale out process for logstash.

I am planning to have 2300 evt/sec events into logstash. This number includes both from filebeat and syslog of network device. But number of logs will be planned to increase more.

I understand that your options for performance increase for logstash will be increasing input queue (beats or syslog) or logstash itself. However, these option makes you change the configuration of filebeat side (increase worker and end up needing more cpu cores) and the network device side for changing desitination. I believe that idea of Message Queue (redis) can be used which can simplify the configuration of filebeat and network device

My question is how do you usually send logs from network device which was directly sending to logstash into message queue?

2

Network devices (I'm assuming you're talking about e.g. switches and routers) typically send events via syslog and don't support message queues. What you can do is have one or more syslog receivers that don't do anything with the events except push them onto a queue, and then you can have one or more Logstash instances that pull from this queue and process the events.

3

@magnusbaeck

Ok. Thanks.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.