Logstash to read log files from remote Windows network drives


(Maxim Gueivandov) #1

Hello,

I'm new to ELK, and I would like to set up a solution to index Microsoft IIS and applicative .NET logs with ES.

I'm aware about different approaches:

  1. [app servers: log files ➔ Logstash] ➔ [collecting server: Redis ➔ Logstash] ➔ [ES cluster: ES ➔ Kibana]
    The con of this method is to having to install, configure and maintain a logstash instance on each Windows server producing logs

  2. [app servers: log files ➔ Filebeat] ➔ [collecting server: Logstash ➔ Redis ➔ Logstash] ➔ [ES cluster: ES ➔ Kibana]
    The con of this method is that currently filebeat does not support multiline log entries, and my .NET apps produce multi-line exceptions. I'm not sure how the intermediate logstash+redis+logstash is to be configured to handle this.

So I thought, maybe given that Logstash is able to collect log data without filebeat or any other forwarder by itself (please correct me if I'm wrong), I might try the following:

[app servers: log files] ➔ [collecting server: Samba-mapped network drives ➔ Logstash ➔ Redis ➔ Logstash] ➔ [ES cluster: ES ➔ Kibana]

In that hypothesis, I won't need to install a Logstash instance on each app server. The central logstash instance (or multiple instances) would fetch files (using Samba-mapped network drives) and apply the multiline codec before pushing log entries to Redis.

Is that a sound architectural choice? What are the possible limitations of this approach?

Thanks a lot!

mguiwin


(Mark Walkom) #2

If it works for your needs the yes, go for it!

Just remember that network based file systems can be slow.


(system) #3