Creating Headers (with key value) in kafka offset using logstash Configuration

Elastic Stack Logstash
1

Below is my config i had written to create headers (key & value) in my logstash config , but still it does not reflect to me in headers under offset tool.

input {
    elasticsearch {
    hosts => ["localhost"]
    index => "some_index*"
    query => '{ "sort" : [ {"@timestamp" : {"order" : "asc"}} ], "query": { "range": { "@timestamp": {"gte": "now-1m","lt": "now"} }}}'
    size => 1000
    docinfo => true
    schedule => "*/2 * * * *"
    user => "elasticsearch"
    password => "password"
   }
}
filter {
 mutate {
  add_field => {
          "[@metadata][header_key1]" => "Hello World"
                "[@metadata][header_key2]" => "abc"
}
}
}

output{
  kafka {
                bootstrap_servers => "kafka-server1:9096"
                topic_id => "INT_TEST"
                security_protocol => "SASL_SSL"
                sasl_mechanism => "SCRAM-SHA-512"
                ssl_endpoint_identification_algorithm => "https"
                sasl_jaas_config => "org.apache.kafka.common.security.scram.ScramLoginModule required username='abc' password='password';"
                client_id => "CLUSTER"
                key_serializer => "org.apache.kafka.common.serialization.StringSerializer"
                value_serializer => "org.apache.kafka.common.serialization.StringSerializer"


               codec => json {}
        }}

Please help how we can configure headers for kafka topic in logstash configuration

2

@metadata is a special field to be used in a Logstash pipeline only, they are not present in the output document.

If you want those fields in the output document you need to use other name.

3

Could you please share some sample configuration to create header in kafka topic under my logstash config

4

Note:- We are using logstash-integration-kafka (10.12.0-java) plugin. Still we are not able create or write payload/value to Headers.

5

I don't think this is supported, there is nothing specific about headers in the kafka output documentation.

And there is this old github issue asking for it to be implemented, but the issue is still open.

Apparently you can read headers but not write it.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.