We're using Logstash to write event data to a Kafka queue. The consumer of this queue expects an Avro binary blob (or fragment, the terminology seems to differ sometimes). It seems however, that using the Avro codec provided by Logstash for the Kafka output adapter, results in a base64 encoded version of the binary fragment being produced to Kafka. This is also clear from the source code, see:
We're wondering:
Why this base64 encoding is added (what is its use);
Why this is not configurable in the pipeline conf. I would have expect an option like 'base64 => true/false', but it's not there.
And maybe, if I ask nicely, if this configuration option could be added (and if these forums are then the proper place to request it)?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.