Multiple filebeat instances or not?

Elastic Stack Beats
1

I have a use-case where i can have multiple ES clusters running. I want to be able to send logs from multiple sources to 2 ES clusters.

  • 1st gets business critical logs
  • 2nd gets all common logs, VM specific

This way we can reduce the processing and footprint of a lot of ES clusters except for the 1 (which gets all common logs).

From my understanding, there is no way today to send logs to multiple logstash outputs from 1 FB (v1.2.2) instance. One possible way could be to run 2 multiple instances of FB.

Is that the best way to achieve this ? If yes, can someone explain how can i go about it and what are some of the variables that i'd need to change in order to get this to work

I'm open to suggestion and/or better way to achieve the same.

Thanks!

2

I use to instances of filebeat for this. I run 2 servies, filebeat.service and filebeat2.service. The second one takes its configureation from /etc/filebeat2 instead of /etc/filebeat. Just configure in the service definition that it will read the new config, and it will work just fine. I also use two binaries, renamed the copied one to filebeat2.

3

One thing you have to make sure is that the two filebeat instances use different registry files.

4

I forgot to mention that, but that is the most important indeed! :slight_smile:

5

Thanks guys..let me give that a shot. But any idea if this is a feature will be implemented anytime soon in FB natively.

6

All I can say is that it is not on the short term roadmap, means for the 5.0 release.