Filebeat: How to create multiple instances/pipelines on same VM

[albus]

Hello Elastic Community,

I hope everybody is doing great.

I am facing a problem and I cannot find any documentation or blog related to it. Problem statement is: Is there a possibility to run multiple instances of Filebeat as a different process from different config files ? Each config file will read data from separate path and ship it to separate output. (Each instance should be completely independent of each other.)

One solution I can think of is to use

/usr/bin/filebeat -c /path/to/each _filebeat_config

and define a service for each process.

But I am looking for something a little more "official".

Can anyone help me out by providing a different perspective or a documentation / blog I might have missed.

Thanks.

[leandrojmp (Leandro Pereira)]

@albus

But I am looking for something a little more "official".

It is possible to run multiple instances of filebeat on the same server, but I don't think there is any official documentation on how to do it because it is more an infrastructure issue than a Filebeat issue.

What you find in the documentation is a note telling that when running multiple instances in the same server, they need to use a different data path, so it is not just pointing to a different config file, but you need to configure it to use a different data path.

So you would need to have a different filebeat.yml file for each instance using a different path.data.

[albus]

noted.
is it possible to parse multiple configuration files through -c option in filebeat ?
like:

filebeat -c conf1.yml -c conf2.yml ...... and so on.

[stephenb (Stephen Brown)]

@albus

filebeat -c conf1.yml -c conf2.yml ...... and so on.

No

If you want to run more than 1 filebeat create separate directories and the use the tar.gz distribution in each that is the easiest/ safest way.

[albus]

Thanks.
I am guessing I will have to provide different data and home paths for different instances for complete isolation.
Is that correct ?
Or the default paths (except config) will do the trick ?

[stephenb (Stephen Brown)]

@albus

I am guessing I will have to provide different data and home paths for different instances for complete isolation.
Is that correct ?
Or the default paths (except config) will do the trick ?

Little unclear what you are asking, so to be Precise if you put tar.gz distributions in their own directories and start from the command ./filebeat the data paths will be relative to each directory so defaults will/should work... I believe

Quick test should show you...