# How to get kafka metadata from a confluent kafka topic

**URL:** <https://discuss.elastic.co/t/how-to-get-kafka-metadata-from-a-confluent-kafka-topic/383844>\
**Category:** Logstash\
**Created:** [December 3, 2025, 12:20pm UTC](https://discuss.elastic.co/t/how-to-get-kafka-metadata-from-a-confluent-kafka-topic/383844 "2025-12-03T12:20:52Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![venkatkumar229](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkatkumar229/32/104663_2.png) [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)\
**Post date:** [December 3, 2025, 12:20pm UTC](https://discuss.elastic.co/t/how-to-get-kafka-metadata-from-a-confluent-kafka-topic/383844/1 "2025-12-03T12:20:52Z")

</div>

Hi,

I am reading the data using logstash from a confluent kafka topic and I am trying to get the complete kafka metadata including header etc into my data. I have tried the below config but still i am unable to see the metadata included in the output. Could you please help here.

I have checked the payload from kafka and this is my sample payload:

{"partition\_id":"1","key":{"type":"STRING","data":"xxxxxxxxxxxxxxx"},"headers":[{"name":"original\_filename","value":"xxxxxxxxxxxxxx"},{"name":"project","value":"xxxxxx"},{"name":"app\_source","value":"xxxxxxx"}],"value":{"type":"JSON","data":{"documents":[{}]}}}

Config:

```auto
input {
  kafka {
    bootstrap_servers => <kafka_server_list_redacted>
    topics => ["test_topic"]
    group_id => "xxxxxxxxxxxxxx"
    security_protocol => "SASL_SSL"
    sasl_mechanism => "PLAIN"
    sasl_jaas_config => "xxxxxxxxxxxxxxxxxx"
    codec => json {}
   decorate_events => extended
  }
}
filter {
}
output {
  elasticsearch {
    hosts => <elasticsearch_hosts_redacted>
    index => "%{[@metadata][kafka][lc_topic]}-%{+YYYY.MM.dd}"
  }
}

```

When i run this config i am getting only the data present in the "value":{"type":"JSON","data":{"documents":[{}]} but not the other data which i want.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 3, 2025, 5:47pm UTC](https://discuss.elastic.co/t/how-to-get-kafka-metadata-from-a-confluent-kafka-topic/383844/2 "2025-12-03T17:47:18Z")

</div>

I cannot test it, but my understanding from the [code](https://github.com/logstash-plugins/logstash-integration-kafka/blob/1cc3a65801fad82dbc9eca0d09ecf9bc690e20b2/lib/logstash/inputs/kafka.rb#L402) is that if you set decorate\_events to extended then everything gets set in the [@metadata][kafka] field. Almost universally, outputs don’t do anything with [@metadata] and the elasticsearch output will not send it to elasticsearch. If you want to get that data into elasticsearch you could try

```auto
mutate { rename { "[@metadata][kafka]" => "[kafka_metadata]" } }

```

---

<div class="post-metadata">

**Author:** ![venkatkumar229](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/venkatkumar229/32/104663_2.png) [@venkatkumar229](https://discuss.elastic.co/u/venkatkumar229)\
**Post date:** [December 4, 2025, 5:35am UTC](https://discuss.elastic.co/t/how-to-get-kafka-metadata-from-a-confluent-kafka-topic/383844/3 "2025-12-04T05:35:20Z")

</div>

Thankyou So much @Badger . It worked and now i am able to see the metadata in the “kafka\_metadata” object.