Hello, I have recently been evaluating Filebeat and Vector as log collectors for gateway logs. Filebeat has strong performance, but the only issue is that it does not support multi-pipeline or multi-sink modes. If I want to collect different log paths from the gateway and send them to different Kafka topics, I would need to configure multiple sets of Filebeat services, which doesn't quite meet my expectations.
Although Vector can support this scenario by combining multiple sinks and sources into different pipelines, I would like to ask if there are any plans for Filebeat to support this feature?
Having multiple outputs in beats is something that was requested multiple times, but Elastic already mentioned that they have no plan to add it, you can check it here.
You do not need multiple filebeat services, this use case is already supported and is the example in the documentation for the Kafka output, you can check it here.
You just need to add a custom field to each input you have in filebeat, and use this custom field in the output.
Thank you for your response. While outputting to multiple Kafka topics can address several issues in this scenario, it may be challenging to support different sinks such as Kafka, RocketMQ, etc., if needed. Regardless, I greatly appreciate your input.
Yeah, this is another issue, Filebeat outputs are pretty limited, it was built to be used with Elastic tools and besides Elasticsearch and Logstash we have outputs to Redis and Kafka basically because both are commonly used as message buffers in deployments of Elasticsearch.
I would not expect new outputs destinations to be added nor multiple outputs being supported.
Vector is way more flexible than Filebeat and depending on the use case it is more flexible than Logstash also.