SOLVED - Logstash http output to Kafka REST Proxy

Elastic Stack Logstash
1

Wanted to share a solution that I found to Logstash http output to Kafka-REST-Proxy and Logstash to Confluent kafka rest proxy. Was running into the same issue trying to use the http output to write a confluent Kafka REST proxy, but couldn't find any answers. Managed to figure it out. Wanted to share the solution that worked for me here (and probably future me will be most thankful if this helps nobody else!).

output {
    http {
        http_method => "post"
        url => "https://example.host/kafka-rest/customer/topics/TOPIC_HERE?apikey=X"
        headers => ["Accept", "application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json"]
        content_type => "application/vnd.kafka.json.v2+json"
        format => "message"
        message => '{
                        "records":[
                            {
                                "key": "%{key}",
                                "value": {
                                    "f1": "%{f1}",
                                    "f2": "%{f2}"
                                }
                            }
                        ]
                    }'
    }
}

Above is assuming that key, f1, and f2 fields exist in the event (whether via input or created by add_field, or whatever).

Also, this Kafka REST POST doc can be helpful: Confluent REST Proxy API Reference | Confluent Documentation

Was using Logstash 6.5.1 (don't judge), but imagine would work similarly on newer versions as well.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.