Is it reasonable to use multiple instances of Filebeat (Windows) on a production system?

Elastic Stack Beats
1

To whom it may concern,

Background: Currently in our windows servers we have different applications that belong to different clients. Therefore, we want to create different indexes with different ILM policies since each client will have different retention times and policies. Therefore, it is essential that each client can have its own policies.

Problem statement: We have proposed a simple architecture, through which Filebeat is directly ingested in Elastic (Elastic Cloud), without going through Logstash. However, we have encountered the following problem: in Filebeat we can configure different patterns for the index, but it is only possible to configure an index template, and therefore, it is only possible to associate an ILM policy to this index template. The problem is that we cannot associate several index templates, one for each client on a single Filebeat configuration.

Possible solution: After researching, I read that there would be no problem in having different instances of Filebeat on Windows. This way, each Filebeat instance would manage the logs of each client. The problem with this solution is scalability; it is not sustainable or maintainable to have 10 or 15 Filebeat instances, as the operation would not be efficient.

Any solutions? I think the only solution is to include Logstash in the equation: send everything from Filebeat to Logstash, and manage the creation of indexes, index templates and ILM policies in Logstash.

2

Hello @ruben.crespo.cano,

You can use single Logstash and can ingest these Logs in different indices and apply ILM on the indices.
According to me it would be a good one inspite of having multible beats on one server.

Use single beat and one logstash server.

PS: this is according to me

Regards

3

How do you identify the different applications, is there something in the logs?

4

Hello @warkolm,

No, each client will store their log and metric files in different locations / paths.

For example:

  • C:\ProgramData\Observability\CLIENT_1\logs
  • C:\ProgramData\Observability\CLIENT_2\logs
  • C:\ProgramData\Observability\CLIENT_3\logs
5

I would use a single instance, and define an input per path, also adding a tag to identify which customer the path belongs to.

That way you can then use that as a variable in the output, so that logs go to their own ILM setup.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.