Multiple Filebeat instance

Elastic Stack Beats
1

I am using Filebeat 5.1 on Centos Linux and have following constraints =>

  1. need to run multiple Filebeat instances on same hosts.
  2. Each instance shall work with its own config yml.
  3. Start and stop individual Filebeat instances without affecting other instances.

Can anybody provide more details how to achieve any of the above requirements ?

Also, it will be helpful to know whether there is any API by which I can start/stop Filebeat.
Using /etc/init.d/filebeat restart command somewhat not really good way to start/stop process on production hosts.

Thanks

2

Why do you need more instances? What is your use case?
I also investigated how to have multiple instances. No result. Finally I was able to use only one for all my use cases.

3

Running multiple instances should not be a problem. Just make sure they use different registry files.

4

@ruflin
With default Filebeat interface (/etc/init.d/filebeat on Linux) , I can not create multiple instances.
Start / Stop arguments to script start/stop only one instance. Also, it always use default YML config (/etc/filebeat/filebeat.yml) even if I use different config through -c argument.

How to start multiple instances here ?

5

@Vid
I want to monitor log files for multiple applications producing different types of logs in production.
Logs for some apps need to be streamed to Logstash and for others to Kafka.
It makes easy to manage my requirements with multiple Filebeat instances .
If one Filebeat instance stops/slows down, others shall not be affected.

6

which init system is your distribution using? The /etc/init.d/filebeat file is mostly used as fallback if the distribution is not using systemd. That is you likely have edited the wrong file?

7

@ruflin
I am using Filebeat 5.1 on Centos Linux .
To run multiple instances of Filebeat , I use binary of Filebeat (located at /usr/share/filebeat/bin/filebeat)

Here is command to run : " /usr/share/filebeat/bin/filebeat -c & "
I am not using default wrapper to run Filebeat.
As I run it in background, I get PID for filebeat process. I use this PID to stop/kill filebeat process whenever required.

Is this correct way to stop filebeat by killing its process ?
Or there is any other way to run multiple instances

1 Like
8

This should work. Did you hit any issues?

1 Like
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.