Best practices to introduce a syslog server between log sources and elasticsearch

[jawadak]

Hi, I have a requirement to store the raw logs collected from the endpoints. So I am planning to introduce a syslog server between endpoint and elasticsearch. The general deployment would be,

Log sources -> rsyslog/syslog-ng -> elastic

I am planning to have a 3node elastic cluster. So my doubts are:

1. which syslog server is better: syslog-ng or rsyslog?
2. Can I install the syslog server in one of the nodes itself?

Would be grateful if you can share your inputs.

[spinscale (Alexander Reelsen)]

Instead of syslog, have you considered using filebeat, which can read log data and send it directly to Elasticsearch? This way the data would be collected on the nodes where the events are generated and shipped directly to Elasticsearch without any component inbetween.

[jawadak]

But for a compliance purpose, we need the raw logs. So If I use winlogbeat for windows and send to elastic, there is some processing on data.

So I am looking for a solution to that. And the idea is to send all the logs to syslog server and then use filebeat to ship the logs.

Also I am looking for a solution to send windows event logs to the syslog server.

[spinscale (Alexander Reelsen)]

I'm sorry but I do not understand your statement. The raw log message is also part of the data, it is just split into more field to make it easier to search for. What exactly is the problem here with compliance (I am not doubting there is one, just want to understand it better).

For the windows event logs there is winlogbeat, but again it is sending the data directly to elasticsearch, so not sure if you're after that.

[jawadak]

It's not only compliance, but also for a forensics purpose. We need untouched raw logs which haven't undergone any processing.