How to setup multiple input and output in Filebeat config?

Elastic Stack Beats
1

I need to have 2 set of input files and output target in Filebeat config.

My current filebeat.yml config looks like this:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /path/to/log-1.log
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
output.logstash:
  hosts: ["myLogstashServer1:5044"]

I want to add another log path and target to the config. Filebeat listen both log-1.log and log-2.log, but log-1.log only output to myLogstashServer1:5044 , log-2.log only output to myLogstashServer2:5044 :

  • /path/to/log-1.log --> myLogstashServer1:5044
  • /path/to/log-2.log --> myLogstashServer2:5044

Is it possible? If yes, how to do that?

2

I believe Filebeat only supports a single output. Why can you not send the data to a single Logstash instance?

3

Because the new logstash server is for vendor to monitor their application log.

In this case, what is the best approach? Can I use Grok filter plugin to whitelist logs that I want?

4

A Logstash instance can use conditionals and multiple outputs. What are the requirements around this solution?

5

I want to filter logs by their path.
For example, logstash server 1 only process logs from /var/log/app-1/*.log

6

Is the requirement that the vendor should be able to provide their own config file for processing their data? Is the vendor running their own Logstash instance?

7

Correct, so I want to set the filtering in the existing Logstash instance.

8

Have you looked into using multiple pipelines within a single Logstash instance? You could have one pipeline receiving all data from Beats and then use conditionals and pipeline to pipeline communication to send data to specific processing pipelines depending on the origin.

You can also have one Logstash instance receive all events and then based on conditionals forward data via Lumberjack to another Logstash instance.

9

Thanks for your suggestion, but for some reason, we are not allowed to share the Logstash instance..

10

Then use a Lumberjack output plugin combined with a beats input plugin to send data from one instance to another.

11

Sorry, I don't get it. I am new to elastic products, can you show the data flow between the servers of your solution? Thanks a lot!

12

Filebeat ---> [Beats input] Logstash [Lumberjack output] ---> [Beats input] Logstash [Elasticsearch output] ---> Elasticsearch

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.