Filebeat system module to kafka topic

Elastic Stack Beats
1

Hi all

I have a setup currently sends snort logs to kafka topic and to logstash for ingest. Just wondering if is possible to send Filebeat System module logs to kafka topic ?

Regards
Yu Feng

2

Hey @ismyhairnice, welcome to discuss :slight_smile:

Yes, you can configure Filebeat to send any logs to a kafka topic, no matter the source of these logs.
To collect the system logs you need to enable the system module.

3

Thanks for your answer Jaime. I have a question regarding logs sent to the kafka output.

I would like to design a solution that buffers log sent via Beat agents. Actually if I understand well the beat modules use the pipeline of the Ingest Node and that's how the processing is done. The drawback is that we may lose some logs if our es cluster in unhealthy. I would like to buffers these logs into kafka before sending them to es (Ingest Node pipeline). Do you have an architecture in mind to do that ? I would like to avoid to use Logstash pipeline. Does logstash indexer will read kafka topics et call the appropriate Ingest Node pipeline ? Do you know another tool that does that ?

Thanks very much.

4

Logs sent to Kafka by Beats can be collected with Filebeat with the Kafka input, and then sent to ES. There is no need to use Logstash.

So you can have: Beats -> Kafka -> Filebeat -> ES.

Regarding buffering, apart of the previous architecture, each beat has also a local queue that keeps some events in memory in case the output is not available. It can be also configured to spool to disk. Take a look to the documentation for the internal queue.

5

Awesome ! Thank you very much Jaime.

Will try internal queue (with spool queue in configuration) and if it works well we no longer need Kafka in our architecture :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.