Rsyslogd replaces with Logstash and/or Filebeat?

[jasony (Jason)]

hello,

I am managing one cluster and current service-flow is as below.

SYSLOG -> Rsyslogd (store data into disk) -> Logstash (read from disk and transfer to Kafka) -> Kafka -> Logstash -> Elasticsearch

Problem is current Rsyslogd is not really stable, and I would replace with Logstash or Filebeat.

Option#1:

If i will go with Logstash, service-flow will be like below.

SYSLOG -> Logstash (directly transfer to Kafka, then store into disk as backup) -> Kafka -> Logstash -> Elasticsearch

Option#2:

If i will go with Filebeat, service-flow will be like below.

SYSLOG -> Filebeat (store data into disk) -> Logstash (read from disk and transfer to Kafka) -> Kafka -> Logstash -> Elasticsearch
OR
SYSLOG -> Filebeat (directly transfer to Logstash) -> Logstash (transfer to Kafka) -> Kafka -> Logstash -> Elasticsearch

Can you please advise which option is better and what i need to consider more for stability?

Thank you!

[staodd (Staale)]

I find it weird that you say rsyslogd is unstable. Usually when it's configured properly it's stable and does what you tell it. You say little about your environment, so it's hard to make a qualified response. What load do you have (msg/sec). What kind of network design? How many servers.. etc etc. To minimize disk IO you might wanna look at having rsyslogd forward directly to logstash instead of having logstash read from files. Then you can keep logstash at one server, and just use rsyslogd to forward msgs.