How can I get data from Filebeat to Flume?

Elastic Stack Beats
1

I can't see a simple way to get data from Filebeat into Flume. Any ideas? Maybe due to Filebeat's youth, there just aren't very many generic ways (HTTP, TCP) to get data out of Filebeat?

Filebeat looks like a rare combination of . . .

  1. Able to read events recorded in log files while Filebeat wasn't running

  2. Compiled or Java

  3. Actively developed

Those qualities made me think it would be a good fit for my go-to log file collector for a new system I'm designing.

The main requirement that seems to eliminate so many log collectors is the ability to "catch up" and read all logged events even if they were recorded while the log collector is offline. Due to audit requirements, this system can't miss events due to network connection problems or any other reason; all logged events must be persisted in long-term storage. I'm trying to find a log collector that will do so and therefore avoid having to monkey with how each and every system, app, OS manages log file rotation, etc.

2

Checking the default source types supported by flume, the spool or kafka source might help getting the logs.

3

@ElasticCrank Perhaps you can use Logstash as a middle man as LS has many more outputs.

4

This topic was automatically closed after 21 days. New replies are no longer allowed.