Where to drop jar containing custome plugin or encoder like custom Kafka message decoder

How to use custom decoder. What is the right location of a jar file containing this custom decoder. I

You mean codec?

Not codec. I meant decoder_class . According to the document the default value is kafka.serializer.DefaultDecoder . I replaced it with a custom java class and tried dropping the jar file containing this class at different locations in my logstash home directory. It fail to find the class. ( logstash version - 2.1.1).

I don't think we ever nailed this down for the feature. You can find the
Kafka jar and drop it in that directory. Not ideal but it should work...

Thanks Joe for the reply. I had tried this - adding this in kafka_2.10- . Got the following error
Couldn't find any input plugin named 'kafka'. Are you sure this is correct? Trying to load the kafka input plugin resulted in this error: cannot load Java class kafka.consumer.ConsumerIterator

I verified the missing file the above error mentions and related files are there in the repackaged jars.
My logstash version is 2.1.1. and java env is 1.8.0_31. Can the java version be an issue in case this and other original jar files in the distribution were compiled and archived in java 1.7.x environment? Any other reason ?

I found an easy solution .
Its applicable to i) logtash 2.1 & ii) if you have maven build system.
Adding maven coordinates (groupId, artifactId and version) as it appears in local maven repository's pom file for all my jar dependencies in "./vendor/bundle/jruby/1.9/gems/jruby-kafka-1.4.0-java/lib/jruby-kafka_jars.rb" did the trick.

1 Like

Thanks for the tip!