Using output Kafka Codec Avro with KafkaAvroSerializer (schema_registry_url issue)

Elastic Stack Logstash
1

Hi guys,

Our purpose is to post messages to event bus Kafka using Avro.

We added all required jars libraries in LS_HOME/vendor/jruby/lib/
And Logstash does recognize KafkaAvroSerializer and all other required classes fine.

The Issue is that logstash needs to specify the SCHEMA_REGISTRY_URL

[logstash.outputs.kafka   ][main] Unable to create Kafka producer from given configuration {:kafka_error_message=>org.apache.kafka.common.KafkaException: Failed to construct kafka producer, :cause=>io.confluent.common.config.ConfigException: Missing required configuration "schema.registry.url" which has no default value.}

But this property could not be passed to the kafka output plugin

    kafka {
            codec => avro {
                         schema_uri => "path/to/Schema.avsc"
                         tag_on_failure => true
                     }
            
            retries => 10
            bootstrap_servers => "xxx:9092"
            topic_id => "xxx"
            acks => "all"
            message_key => "xxx"

            key_serializer=> "io.confluent.kafka.serializers.KafkaAvroSerializer"
            value_serializer => "io.confluent.kafka.serializers.KafkaAvroSerializer"
        }

Here is how our "vendor/lib" looks like

avro-1.10.0.jar
common-config-5.5.1.jar
jni/
jruby.jar
kafka-avro-serializer-5.3.0.jar
kafka-clients-2.5.0.jar
kafka-schema-registry-client-5.3.0.jar
kafka-streams-avro-serde-5.5.1.jar
ruby/

Any one could help with this issue please ?

2

Take a look at this thread.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.