Two filebeat instances

Elastic Stack Beats
1

can i get two filebeat instances working at the same time, one sending to elasticsearch and the other to logstash?

i have lots of messages being parsed by logstash (postfix, cyrus) and it would be too much trouble migrating everything.

on the other hand, i like the [simple] way filebeat works with elasticsearch, installing kibana dashboards and so on.

2

Yes, you can run two filebeat instances in parallel. Make sure they use a different data directory so they have 2 different registrar files.

3

could this be a feature in next releases?

configuring the output in the filebeat.prospectors section instead of a global, yml wide, output. section?

so we wouldn't have to run two instances.

4

With 6.0 we removed the support two have 2 outputs enabled in parallel as it was more an accidential feature and caused issues. Interestingly your feature request here is more about routing events to the correct output. If you need to route your events and prefer not to run two beats binaries, I would recommend you to put Logstash in the middle for the routing.

5

i am currently using logstash to parse logs (apache, cyrus, nginx, postifx, system).

but i like the idea of having each server parsing its own logs and sending it right to elasticsearch, without logstash in de middle.

so, for now, i have to keep logstash parsing complex logs like cyrus, but for everything else filebeat is the right tool for job.

to which version should i downgrade to get the 'accidental feature' that i need?

6

The "accidential feature" of being able to send data to Logstash and Elasticsearch in parallel is in all Filebeat versions previous to 6.x. But I'm not sure if that fully covers your use case as all events are always sent to both outputs.

7

Like yodog, I need to route Project A prospectors (4) to Logstash Port 1 and Project B prospectors (2) to Logstash Port 2. It is odd to me that this common use case is not supported without running another Filebeats instance.

That said, where can I find a how-to guide to run two different Filebeats instances beyond:

Yes, you can run two filebeat instances in parallel. Make sure they use a different data directory so they have 2 different registrar files.

8

i requested a review of this feature here: https://github.com/elastic/beats/issues/6531

maybe if you comment on the issue they realize more people want this and give it a go...

9

@mn_dev There aren't really docs on this but you are right we probably should have some. Here are more details on the data path directory: https://www.elastic.co/guide/en/beats/filebeat/master/configuration-path.html#_literal_data_literal Happy to help here if you hit some issues with it.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.