Kafka topics configuration example

[sykomaniac (Ash Sykes)]

Hi,

Is anyone able to provide me with an example of configuring the topics setting for Kafka output.

I think I can use this to potentially write data points to different topics. E.g. write a mapping of kubernetes --> pods to a pods topic, kubernetes --> nodes to a node topic etc.

Thanks

[kvch (Noémi Ványi)]

How exactly you would like to separate your data? The processor add_kubernetes_metadata adds pod, namespace and labels to your events. (https://www.elastic.co/guide/en/beats/filebeat/current/add-kubernetes-metadata.html) Full list of provided fields: https://www.elastic.co/guide/en/beats/filebeat/current/exported-fields-kubernetes-processor.html

You could use these fields to send your events to different topics as seen here: https://www.elastic.co/guide/en/beats/metricbeat/current/kafka-output.html#_literal_topic_literal

To send every event from a namespace to the same topic you could add this to your Kafka output config:

topic: '%{[fields.kubernetes.namespace]}'

[sykomaniac (Ash Sykes)]

Hi @kvch thanks for the reply.

That's exactly my question. I want to make use of the topics configuration item to send messages to different topics based on some of the kubernetes metadata e.g. any node metrics to a node topic, and pod metrics to a pod topic etc. I just wondered if someone could provide a practical example of this configuration as I can't seem to find one to use as a basis.

Thanks