Non-optional base64 encoding in Avro codec

[robinklaassen]

We're using Logstash to write event data to a Kafka queue. The consumer of this queue expects an Avro binary blob (or fragment, the terminology seems to differ sometimes). It seems however, that using the Avro codec provided by Logstash for the Kafka output adapter, results in a base64 encoded version of the binary fragment being produced to Kafka. This is also clear from the source code, see:

logstash-plugins/logstash-codec-avro/blob/master/lib/logstash/codecs/avro.rb

We're wondering:

1. Why this base64 encoding is added (what is its use);
2. Why this is not configurable in the pipeline conf. I would have expect an option like 'base64 => true/false', but it's not there.

And maybe, if I ask nicely, if this configuration option could be added (and if these forums are then the proper place to request it)?

Thanks in advance,
Robin

[robinklaassen]

We have now forked the Avro codec and made this base64 encoding configurable. See: https://github.com/Rubyan/logstash-codec-avro. Feel free to use!

Usage example:

output {
    stdout {
        codec => avro {
            schema_uri => "schema.avsc"
            base64_encoding => false
        }
    }
}