Filebeat Kafka topic mappings

Hi I am trying to setup filebeat to send events to Kafka and have a requirement that I will need to send events from different apps to a different topic.

I tried to set the configs as mentioned in the documentation(https://www.elastic.co/guide/en/beats/filebeat/7.2/kafka-output.html#topics-option-kafka) but nothing works and it always falls back to the default topic. Could someone please shed more light as the documentation doesn't show any example around this.

My filebeat.yaml is as below.

Case 1: (Doesn't work)

    filebeat.inputs:
    - type: container
      paths:
        - /var/lib/docker/containers/*/*.log
      fields:
        log_type: containerlogs
        group_name: kafka_default_group
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            cluster: ${CLUSTER_NAME}
            matchers:
            - logs_path:
                logs_path: "/var/lib/docker/containers/*/"

    output.kafka:
      codec.json:
        pretty: true
        escape_html: false
      hosts: ['broker1:9092', 'broker2:9092']
    topic: Kafka_default_topic  
    topics:
        topic: logs_topic_0
        when.equals:
          appname: app1
        topic: logs_topic_2
        when.equals:
          appname: app1
       
      partition.round_robin:
        reachable_only: false
    required_acks: 1
    max_message_bytes: 10485760

Case 2: (Doesn't work)

filebeat.inputs:
- type: container
  paths:
    - /var/lib/docker/containers/*/*.log
  fields:
    log_type: containerlogs
    group_name: kafka_default_group

  processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        cluster: ${CLUSTER_NAME}
        matchers:
        - logs_path:
            logs_path: "/var/lib/docker/containers/*/"

output.kafka:
  codec.json:
    pretty: true
    escape_html: false
  hosts: ['broker1:9092', 'broker2:9092']
  topics:
  - topic: '%{[fields.appname]}'
    mappings:
      app1: logs_topic_0
      app2: logs_topic_1
    default: kafka_default_topic
  partition.round_robin:
    reachable_only: false
required_acks: 1
max_message_bytes: 10485760
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.