Filebeat send different files to different output

Elastic Stack Beats
1

Hi,

We have a requirement where we need to parse two different logs using filebeat.

  1. Send the first file content to Kafka
  2. Since the second file has lot of irrelevant information, some basic filtering will be done in filebeat and the rest of the content should be written to another file in the same machine.

I've seen that filebeat was supporting multiple output in 5.6.0, but in 6.x version I see that feature is removed stating that there would be problem with the registry.

Can some one explain why this is not supported in 6.x though we are going to harvest different file and send to different output. And any possible solution without sending the data to logstash since we need the filtered content of second file to be available in that same machine.

2

The reason for multiple outputs are not supported is explained in this issue. Moreover, even in 5.x you could not selective route data to one particular output, all data went to all outputs.

You could run two separate instances of Filebeat. One for sending direct to Kafka. And one for parsing/filtering then writing a file.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.