Kafka topics configuration example

(Ash Sykes) #1


Is anyone able to provide me with an example of configuring the topics setting for Kafka output.

I think I can use this to potentially write data points to different topics. E.g. write a mapping of kubernetes --> pods to a pods topic, kubernetes --> nodes to a node topic etc.


(Noémi Ványi) #2

How exactly you would like to separate your data? The processor add_kubernetes_metadata adds pod, namespace and labels to your events. (https://www.elastic.co/guide/en/beats/filebeat/current/add-kubernetes-metadata.html) Full list of provided fields: https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-kubernetes-processor.html

You could use these fields to send your events to different topics as seen here: https://www.elastic.co/guide/en/beats/metricbeat/current/kafka-output.html#_literal_topic_literal

To send every event from a namespace to the same topic you could add this to your Kafka output config:

topic: '%{[fields.kubernetes.namespace]}'

(Ash Sykes) #3

Hi @kvch thanks for the reply.

That's exactly my question. I want to make use of the topics configuration item to send messages to different topics based on some of the kubernetes metadata e.g. any node metrics to a node topic, and pod metrics to a pod topic etc. I just wondered if someone could provide a practical example of this configuration as I can't seem to find one to use as a basis.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.