Filebeat: Unable to create dynamic Kafka topic name using Kubernetes metadata

Elastic Stack Beats
,
1

I am trying to push Kubernetes logs to the Kafka cluster but while pushing logs I want to create a dynamic topic for all the corresponding Kubernetes namespaces.

I am using the below configuration but getting errors while creating topics.

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-prospectors` configmap:
        path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        reload.enabled: true
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: true
    processors:
      - add_cloud_metadata:
      - drop_fields:
            fields: ["host"]
    output.kafka:
      hosts: ['host:9092']    
      topic: '%{[kubernetes.namespace]}'
      partition.round_robin:
         reachable_only: false
      required_acks: 1
      compression: gzip
      max_message_bytes: 1000000

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  kubernetes.yml: |-
    - type: container
      symlinks: true
      paths:
        - '/var/log/containers/*.log'
      multiline.pattern: '^{'
      multiline.negate: true
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
        - drop_fields:
            fields: ["host"]
        - rename:
            fields:
             - from: "kubernetes.labels.app"
               to: "kubernetes.labels.app_name"      
---

**I am getting error as: **

|2021-09-28T06:12:49.973Z|ERROR|kafka/client.go:144|Dropping event: no topic could be selected|
|---|---|---|---|
|2021-09-28T06:12:49.973Z|ERROR|kafka/client.go:144|Dropping event: no topic could be selected|
|2021-09-28T06:12:49.973Z|ERROR|kafka/client.go:144|Dropping event: no topic could be selected|

I also tried with below syntax but same issue :
topic: '%{[fields.kubernetes.namespace]}' or topic: '%{[kubernetes][namespace]}' or topic: '%{[kubernetes.namespace]}'

Any help?

2

I got the issue:

updated config:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  kubernetes.yml: |-
    - type: container
      symlinks: true
      paths:
        - '/var/log/containers/*.log'
      multiline.pattern: '^{'
      multiline.negate: true
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            default_matchers.enabled: false
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_fields:
            fields: ["host"]
        - rename:
            fields:
             - from: "kubernetes.labels.app"
               to: "kubernetes.labels.app_name"

topic: "tools-%{[kubernetes.namespace]}"

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.