# Filebeat: Unable to create dynamic Kafka topic name using Kubernetes metadata

**URL:** <https://discuss.elastic.co/t/filebeat-unable-to-create-dynamic-kafka-topic-name-using-kubernetes-metadata/285316>\
**Category:** Beats\
**Tags:** docker, filebeat\
**Created:** [September 28, 2021, 7:01am UTC](https://discuss.elastic.co/t/filebeat-unable-to-create-dynamic-kafka-topic-name-using-kubernetes-metadata/285316 "2021-09-28T07:01:01Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![arunporwal](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arunporwal](https://discuss.elastic.co/u/arunporwal)\
**Post date:** [September 28, 2021, 7:01am UTC](https://discuss.elastic.co/t/filebeat-unable-to-create-dynamic-kafka-topic-name-using-kubernetes-metadata/285316/1 "2021-09-28T07:01:01Z")

</div>

I am trying to push Kubernetes logs to the Kafka cluster but while pushing logs I want to create a dynamic topic for all the corresponding Kubernetes namespaces.

I am using the below configuration but getting errors while creating topics.

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-prospectors` configmap:
        path: ${path.config}/prospectors.d/*.yml
        # Reload prospectors configs as they change:
        reload.enabled: true
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: true
    processors:
      - add_cloud_metadata:
      - drop_fields:
            fields: ["host"]
    output.kafka:
      hosts: ['host:9092']    
      topic: '%{[kubernetes.namespace]}'
      partition.round_robin:
         reachable_only: false
      required_acks: 1
      compression: gzip
      max_message_bytes: 1000000

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  kubernetes.yml: |-
    - type: container
      symlinks: true
      paths:
        - '/var/log/containers/*.log'
      multiline.pattern: '^{'
      multiline.negate: true
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
        - drop_fields:
            fields: ["host"]
        - rename:
            fields:
             - from: "kubernetes.labels.app"
               to: "kubernetes.labels.app_name"      
---

```

\*\*I am getting error as: \*\*

```auto
|2021-09-28T06:12:49.973Z|ERROR|kafka/client.go:144|Dropping event: no topic could be selected|
|---|---|---|---|
|2021-09-28T06:12:49.973Z|ERROR|kafka/client.go:144|Dropping event: no topic could be selected|
|2021-09-28T06:12:49.973Z|ERROR|kafka/client.go:144|Dropping event: no topic could be selected|

```

I also tried with below syntax but same issue :  
`topic: '%{[fields.kubernetes.namespace]}'` or `topic: '%{[kubernetes][namespace]}'` or `topic: '%{[kubernetes.namespace]}'`

Any help?

---

<div class="post-metadata">

**Author:** ![arunporwal](https://avatars.discourse-cdn.com/v4/letter/a/f07891/32.png) [@arunporwal](https://discuss.elastic.co/u/arunporwal)\
**Post date:** [September 28, 2021, 8:04am UTC](https://discuss.elastic.co/t/filebeat-unable-to-create-dynamic-kafka-topic-name-using-kubernetes-metadata/285316/2 "2021-09-28T08:04:16Z")

</div>

I got the issue:

**updated config:**

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-prospectors
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  kubernetes.yml: |-
    - type: container
      symlinks: true
      paths:
        - '/var/log/containers/*.log'
      multiline.pattern: '^{'
      multiline.negate: true
      multiline.match: after
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
            default_matchers.enabled: false
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"
        - drop_fields:
            fields: ["host"]
        - rename:
            fields:
             - from: "kubernetes.labels.app"
               to: "kubernetes.labels.app_name"

```

**topic: "tools-%{[kubernetes.namespace]}"**

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 26, 2021, 10:04am UTC](https://discuss.elastic.co/t/filebeat-unable-to-create-dynamic-kafka-topic-name-using-kubernetes-metadata/285316/3 "2021-10-26T10:04:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.