Multiple Filebeat Instances on Windows Hosts

[B4S71]

Hi there,

We currently run a number of Hosts, Exchange-Servers at that.

For Problem identification, we require two kinds of logs: Exchange Servers generate IIS-Logs, which are useful for getting return codes over user & time. These codes are really the default IIS format, and for that reason we believe using the beats-iis module would be the perfect way to go.

However, there is another log which seems quite useful, namely the MessageTracking.log
MessageTracking is basically a CSV. And while the fields are well defined, it might just happen to have (quoted) commas within them, which makes pattern-based filtering a pain, really.
Now Logstash's CSV filter might be able to help me out with that, but then again, that requires me to split those different inputs up into two outputs.

As far as I understand Filebeat does not support multiple outputs (and for a good reason) so the Idea would be to run multiple filebeat instances as a windows service, each of them being responsible for it's dedicated log file.

What I need to understand is how to configure both filebeat instances correctly, in order to have them fully independent from each other.

[Suhas_K (Suhas)]

Hey @B4S71 as of my knowledge we can user multiple instance of filebeat on one single server. I have used multiple filebeats on same machine to push data to one single Logstach instance which has multiple pipelines. Multiple pipelines

As your query is What I need to understand is how to configure both filebeat instances correctly, in order to have them fully independent from each other.

Your both the filebeat.yml should have a proper Logstach output

filebeat.yml will some what look like this

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5042"]
#The port number should be mapped with the input of your logstach 

abc.conf file

input {
   beats {
      client_inactivity_timeout => 1200
      port => "5042"
   }
}

filter {

}

output{

}

I hope i was able to answer your query.

Thanks

[B4S71]

Hi @Suhas_K

As for how to connect beats to logstash, as well as running mutliple logstash pipelines, that is no problem at all.

I was much more thinking about the actual beats configuration.

• Does it make sense to share parts of the package?
  • Example: Would I need two copies of the entire package, or does it make sense to use the same executable, while defining two different beats.yml files and define the services to use different configuration yamls.
• are there any default configurations which I would usually leave untouched, but might result in creating issues due to shared resources
  • example: The Data/Log Path needs to be adjusted, I am sure of, but are there any other required configurations?
• Does setting up two filebeat-services produce any other issues?

Thanks for your initial answer, however. I appreciate your help.

Best Regards

[Suhas_K (Suhas)]

Hey @B4S71

• Well, sharing the same packages. I haven't tried it yet but in my setup i'm using different packages for all the filebeats.
  I think giving individual packages makes more sense. That will help you trouble shoot any problems near future
• As you have mentioned that it might result in creating issue due to shared resources for Data/Log path. Initially my concern was same. So ended up creating different packages for all the filebeats.
• I'm running multiple filebeats on the same machine for quite a while now didn't face any issues yet.

Thanks