Multiple filebeat instances or not?

I have a use-case where i can have multiple ES clusters running. I want to be able to send logs from multiple sources to 2 ES clusters.

  • 1st gets business critical logs
  • 2nd gets all common logs, VM specific

This way we can reduce the processing and footprint of a lot of ES clusters except for the 1 (which gets all common logs).

From my understanding, there is no way today to send logs to multiple logstash outputs from 1 FB (v1.2.2) instance. One possible way could be to run 2 multiple instances of FB.

Is that the best way to achieve this ? If yes, can someone explain how can i go about it and what are some of the variables that i'd need to change in order to get this to work

I'm open to suggestion and/or better way to achieve the same.

Thanks!

I use to instances of filebeat for this. I run 2 servies, filebeat.service and filebeat2.service. The second one takes its configureation from /etc/filebeat2 instead of /etc/filebeat. Just configure in the service definition that it will read the new config, and it will work just fine. I also use two binaries, renamed the copied one to filebeat2.

One thing you have to make sure is that the two filebeat instances use different registry files.

1 Like

I forgot to mention that, but that is the most important indeed! :slight_smile:

Thanks guys..let me give that a shot. But any idea if this is a feature will be implemented anytime soon in FB natively.

All I can say is that it is not on the short term roadmap, means for the 5.0 release.

This topic was automatically closed after 21 days. New replies are no longer allowed.