After searching around and reading source of kafka output plugin. I think that i can support avro encoding with Confluent schema registry format by adding some config to the plugin and use io.confluent.kafka.serializers.KafkaAvroEncoder as encoder class.
However, i can't find any way to add relevant jar files so that jruby can find it. I tried to pack it to the gem under /vendor/jar-dependecies/runtime-jars but logstash still complaint that it cant found the needed classes.
[2017-04-18T04:22:35,187][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<NameError: cannot link Java class org.apache.kafka.clients.producer.ProducerConfig (java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory)>, :backtrace=>["org/jruby/javasupport/JavaUtilities.java:54:in get_proxy_or_package_under_package'", "file:/usr/share/logstash/vendor/jruby/lib/jruby.jar!/jruby/java/java_package_module_template.rb:14:inmethod_missing'", "/usr/share/logstash/vendor/local_gems/2a463982/logstash-output-kafka-6.1.5/lib/logstash/outputs/kafka.rb:218:in create_producer'", "/usr/share/logstash/vendor/local_gems/2a463982/logstash-output-kafka-6.1.5/lib/logstash/outputs/kafka.rb:184:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:37:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:282:in register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:293:inregister_plugins'", "org/jruby/RubyArray.java:1613:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:293:inregister_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:302:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:232:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:387:in `start_pipeline'"]}
Also, adding the jars into CLASSPATH does not help.
I cant find any document on how to use java files with logstash plugin. Most things i found are from 1-2 years ago.
Edit: my jars are not avaiable on maven