Not able to read Kafka - Avro Schema messages

Elastic Stack Logstash
14

There are two problems:

  1. Schema Registry API returns the Avro schema in schema object. You expect that the Avro schema is not inside of nested object. You bypassed this problem with defining schema with a file from local disk (it is not ideal, but it works :).

  2. Every message inside of Confluent Platform should have following structure:
    < magic byte > < schema id (4 bytes) > < Avro blob >
    Currently the codec is not prefixing the Avro blob with magic byte nor schema id. So the kafka-avro-console-consumer expects that inside first 5 bytes there is a magic byte and schema id so it cannot pick up the right schema.

The only solution for this is to write a new codec with prefixing Avro blob. Should not be hard, but I am not a Ruby developer :frowning: