Hi I am trying to setup filebeat to send events to Kafka and have a requirement that I will need to send events from different apps to a different topic.
I tried to set the configs as mentioned in the documentation(https://www.elastic.co/guide/en/beats/filebeat/7.2/kafka-output.html#topics-option-kafka) but nothing works and it always falls back to the default topic. Could someone please shed more light as the documentation doesn't show any example around this.
My filebeat.yaml is as below.
Case 1: (Doesn't work)
filebeat.inputs:
- type: container
paths:
- /var/lib/docker/containers/*/*.log
fields:
log_type: containerlogs
group_name: kafka_default_group
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
cluster: ${CLUSTER_NAME}
matchers:
- logs_path:
logs_path: "/var/lib/docker/containers/*/"
output.kafka:
codec.json:
pretty: true
escape_html: false
hosts: ['broker1:9092', 'broker2:9092']
topic: Kafka_default_topic
topics:
topic: logs_topic_0
when.equals:
appname: app1
topic: logs_topic_2
when.equals:
appname: app1
partition.round_robin:
reachable_only: false
required_acks: 1
max_message_bytes: 10485760
Case 2: (Doesn't work)
filebeat.inputs:
- type: container
paths:
- /var/lib/docker/containers/*/*.log
fields:
log_type: containerlogs
group_name: kafka_default_group
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
cluster: ${CLUSTER_NAME}
matchers:
- logs_path:
logs_path: "/var/lib/docker/containers/*/"
output.kafka:
codec.json:
pretty: true
escape_html: false
hosts: ['broker1:9092', 'broker2:9092']
topics:
- topic: '%{[fields.appname]}'
mappings:
app1: logs_topic_0
app2: logs_topic_1
default: kafka_default_topic
partition.round_robin:
reachable_only: false
required_acks: 1
max_message_bytes: 10485760