Where to drop jar containing custome plugin or encoder like custom Kafka message decoder

Elastic Stack Logstash
1

How to use custom decoder. What is the right location of a jar file containing this custom decoder. I

2

You mean codec?

3

Not codec. I meant decoder_class . According to the document the default value is kafka.serializer.DefaultDecoder . I replaced it with a custom java class and tried dropping the jar file containing this class at different locations in my logstash home directory. It fail to find the class. ( logstash version - 2.1.1).

4

I don't think we ever nailed this down for the feature. You can find the
Kafka jar and drop it in that directory. Not ideal but it should work...

5

Thanks Joe for the reply. I had tried this - adding this in kafka_2.10-0.8.2.1.jar . Got the following error
Couldn't find any input plugin named 'kafka'. Are you sure this is correct? Trying to load the kafka input plugin resulted in this error: cannot load Java class kafka.consumer.ConsumerIterator

I verified the missing file the above error mentions and related files are there in the repackaged jars.
My logstash version is 2.1.1. and java env is 1.8.0_31. Can the java version be an issue in case this and other original jar files in the distribution were compiled and archived in java 1.7.x environment? Any other reason ?

6

I found an easy solution .
Its applicable to i) logtash 2.1 & ii) if you have maven build system.
Adding maven coordinates (groupId, artifactId and version) as it appears in local maven repository's pom file for all my jar dependencies in "./vendor/bundle/jruby/1.9/gems/jruby-kafka-1.4.0-java/lib/jruby-kafka_jars.rb" did the trick.

1 Like
7

Thanks for the tip!