Hello,
I'm new to ELK, and I would like to set up a solution to index Microsoft IIS and applicative .NET logs with ES.
I'm aware about different approaches:
-
[app servers: log files ➔ Logstash] ➔ [collecting server: Redis ➔ Logstash] ➔ [ES cluster: ES ➔ Kibana]
The con of this method is to having to install, configure and maintain a logstash instance on each Windows server producing logs -
[app servers: log files ➔ Filebeat] ➔ [collecting server: Logstash ➔ Redis ➔ Logstash] ➔ [ES cluster: ES ➔ Kibana]
The con of this method is that currently filebeat does not support multiline log entries, and my .NET apps produce multi-line exceptions. I'm not sure how the intermediate logstash+redis+logstash is to be configured to handle this.
So I thought, maybe given that Logstash is able to collect log data without filebeat or any other forwarder by itself (please correct me if I'm wrong), I might try the following:
[app servers: log files] ➔ [collecting server: Samba-mapped network drives ➔ Logstash ➔ Redis ➔ Logstash] ➔ [ES cluster: ES ➔ Kibana]
In that hypothesis, I won't need to install a Logstash instance on each app server. The central logstash instance (or multiple instances) would fetch files (using Samba-mapped network drives) and apply the multiline codec before pushing log entries to Redis.
Is that a sound architectural choice? What are the possible limitations of this approach?
Thanks a lot!
mguiwin