I have many different kinds of logs, such as php system mysql etc, on same server, and i need to send them to different output destinations, it's very common use cases, and it's very import!
But WHY can't filebeat suport the feature? Let me start several filebeat daemon to solve it???? So stupid solution......
Filebeat is a good software, but this design is really really stupid!!!
You might be able to run logstash with a pipeline for each kind of log instead of fllebeat. Probably more overhead, but we do it for some things.
The reason of i use filebeat is that it's light, effective and saving resouces, if i chose to use logstash filebeat would be meaningless for existing, i think almost every thing i can do with filebeat can be finished by logstash.
Just hope filebeat can be a better software, to achieve the goal some basic problems should be solved.
@argb What kind of outputs do you want to use at the same time? Redis/ES/Kafka?
Concerning the fact that Filebeat only supports a single output we do understand that this limitation is problematic for some use cases. We currently choose to only support one output for multiples reasons.
Beats is meant to be a lightweight shipper to get logs off your machine as fast as possible, sending to a single output help us with that goal.
In the current design when Filebeat starts, it will create a single pipeline where all events from your different inputs will be sent. All the events are sent to a unique output, because of this we can maximize the batch size of events send to the remote system. Most of the time larger batches results in better throughput.
Having a single output help us with our delivery guarantee. When the output stops accepting messages, we just stop reading the files until the output is back again. The files on disk acts as a queuing system.
When you add multiples outputs to the mix, it adds a lot of complexity to the logic. A few examples:
These are just a few things that explain why beats don't go there by design.
This is the current situation today, we might add support for multiple outputs someday, but we need to have a good story for a few problems that can happen.
In fact, Logstash and Beats can work together to achieve your goal; you can install Filebeat to your server and send events to one or multiple Logstash instances and use Logstash with multiple outputs (or multiple pipelines) to route the events to the different system.
@ pierhugues
Thanks for your patient and detailed replying!
First of all, Filebeat is a good software 
, thanks for beats team's awesome work!
From the view of user, I just hope it become better.
On my opinion, I think Filebeat should have two import work to do: the first, reading/collecting logs data; the second, routing data to specified output. Both the two parts construct the final goal of filebeat: a high performance logs/data shipper.
At present, filebeat provide very good data collecting feature, but he can just chose one partner(output) to work for a whole life, even he has the ability to work together with many partners(output).
I thought about your example problems, I will try to give some suggestions.
*** Are you sending each event to multiples output?**
It should be decided by the configure, at the beginning of the running of beat, it will know how many data routes would be established. It's not a problem. For now, obviously only one route.
*** If a single output is down, what should we do, halt everything or send to responsible server?**
I think every route is a group, not many to many relationships. So what you need to do is just copy what you have done in the current single route to others parallel routes, all the routes are parallel, there are no cross/intersection.
*** Can conditionals be used to send some events to a single output, what are the delivery guarantee of the outputs?**
Same, just do what you have done.
I noticed you said this words "The files on disk acts as a queuing system.", maybe do you need a queue for every route?
I am not familiar with concrete tech stack used by filebeat, I mainly do some business project with PHP, so maybe my understanding is very ridiculous, but i really thought about it seriously .
And now filebeat can work with elasticsearch ingest node, now that it provide this feature, people will try to use it, but it's not as powerful as logstash to handle kinds of formats log data.
So maybe people(for example, me) want to some simple logs directly send to elasticsearch and some complex log send to logstash. If all logs have to be sent to elasticsearch ingest node, it maybe can't satisfy the business requirement; but if all logs have to be send to logstash, why filebeat provide this feature to send logs directly to elasticsearh? It's a very confused problem, or it actually should not be a problem, it just needs a routing system.
For the configure file, an input configure and an output configure should be in a same group or section.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.