Generate Kafka Topic dynamically with Modules [Filebeat 7.5]

Elastic Stack Beats
1

Hi, everyone

I use Filebeat in order to send all my data to Apache Kafka. Moreover, I use parameter fields in order to generate Kafka Topic dynamically. Here you are an example:

filebeat.inputs:

- type: log
  paths:
    - /var/log/mysql.log
  fields:
   service: mysql

- type: log
  paths:
    - /var/log/apache.log
  fields:
   service: apache

filebeat.config.modules:
  path: ${path.config}/modules.d/system.yml
  reload.enabled: false

logging.level: warning
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 4

output.kafka:
  hosts: ["localhost:9092"]
  topic: "%{[fields.service]}"

Is it possible to generate Kafka Topic dynamically with Filebeat Modules?

Thanks in advance,

Regards :vulcan_salute:

:pencil2: Update post

I achieved it by using processors:

filebeat.inputs:

- type: log
  paths:
    - /var/log/mysql.log
  fields:
   service: mysql

- type: log
  paths:
    - /var/log/apache.log
  fields:
   service: apache

filebeat.config.modules:
  path: ${path.config}/modules.d/system.yml
  reload.enabled: false

processors:
- add_fields:
   when:
    equals:
     event.module: system
   fields:
    service: system

logging.level: warning
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 4

setup.template.enabled: false
setup.ilm.enabled: false

output.kafka:
  hosts: ["localhost:9092"]
  topic: "%{[fields.service]}"
2 Likes
2

I think you will have to set 'auto.create.topics.enable=true' on the broker configuration and on the filebeat output define the topic you want to send/create the logs to.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.