Logstash Kafka Input Plugin Throws Errors missing class name (`org.apache.kafka.clients.consumer.ConsumerConfig')

Elastic Stack Logstash
1

I would really like to use this input plug-in but in the time I have spent futzing around with it I could of written something in GoLang.....The problem is that no matter what I do the plugin does not load:

  • Version:
    Logstash -> ii logstash 1:5.6.3-1 all An extensible logging pipeline
    Logstash Input Module -> logstash-input-kafka-8.0.2

  • Operating System:
    Ubuntu Xenial, 16.04

  • Config File (if you have sensitive info, please remove it):
    input {
    kafka {
    bootstrap_servers => "kafka-node:9092"
    topics => ["topic1","topic2","topic3"]
    }
    }

output {
tcp {
host => "10.155.24.23"
port => "5044"
}
}

  • Sample Data:

  • Steps to Reproduce:

  1. Start kafka,
  2. Watch the logs

[2017-10-24T21:23:39,678][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Kafka bootstrap_servers=>"kafka-seed1.us-west-2.ec2.pages:9092", topics=>["prod.edn.wp-msg-fraudengine-transactions", "prod.log.access-prod-POD", "prod.log.access-prod-identity-check-ng", "prod.log.artifactory", "prod.log.auth"], id=>"8eaa3093e674018d54f6b047b26741246ace3ed7-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_3edbae47-994c-4e6a-bc3a-e9e7a643d4f7", enable_metric=>true, charset=>"UTF-8">, auto_commit_interval_ms=>"5000", client_id=>"logstash", consumer_threads=>1, enable_auto_commit=>"true", group_id=>"logstash", key_deserializer_class=>"org.apache.kafka.common.serialization.StringDeserializer", value_deserializer_class=>"org.apache.kafka.common.serialization.StringDeserializer", poll_timeout_ms=>100, security_protocol=>"PLAINTEXT", sasl_mechanism=>"GSSAPI", decorate_events=>false>
Error: uncaught throw missing class name (`org.apache.kafka.clients.consumer.ConsumerConfig') in thread 0x1d6c28e2

  • Version:
    Logstash -> ii logstash 1:5.6.3-1 all An extensible logging pipeline
    Logstash Input Module -> logstash-input-kafka-8.0.2

  • Operating System:
    Ubuntu Xenial, 16.04

  • Config File (if you have sensitive info, please remove it):
    input {
    kafka {
    bootstrap_servers => "kafka-node:9092"
    topics => ["topic1","topic2","topic3"]
    }
    }

output {
tcp {
host => "10.155.24.23"
port => "5044"
}
}

  • Sample Data:

  • Steps to Reproduce:

  1. Start kafka,
  2. Watch the logs

[2017-10-24T21:23:39,678][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Kafka bootstrap_servers=>"kafka-seed1.us-west-2.ec2.pages:9092", topics=>["prod.edn.wp-msg-fraudengine-transactions", "prod.log.access-prod-POD", "prod.log.access-prod-identity-check-ng", "prod.log.artifactory", "prod.log.auth"], id=>"8eaa3093e674018d54f6b047b26741246ace3ed7-1", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_3edbae47-994c-4e6a-bc3a-e9e7a643d4f7", enable_metric=>true, charset=>"UTF-8">, auto_commit_interval_ms=>"5000", client_id=>"logstash", consumer_threads=>1, enable_auto_commit=>"true", group_id=>"logstash", key_deserializer_class=>"org.apache.kafka.common.serialization.StringDeserializer", value_deserializer_class=>"org.apache.kafka.common.serialization.StringDeserializer", poll_timeout_ms=>100, security_protocol=>"PLAINTEXT", sasl_mechanism=>"GSSAPI", decorate_events=>false>
Error: uncaught throw missing class name (`org.apache.kafka.clients.consumer.ConsumerConfig') in thread 0x1d6c28e2

2

Attempted to reinstall everything!

apt-get purge logstash

Then remove all the left over directories, and then re-install

THIS DID NOT RESOLVE THE PROBLEM

3

Attempted to update the plugin...alas...this does not work either. Does not work with the reinstall.

4

This is so overly descriptive:

[2017-10-24T22:21:30,111][ERROR][logstash.inputs.kafka ] Unable to create Kafka consumer from given configuration {:kafka_error_message=>#<NameError: missing class name (`org.apache.kafka.clients.consumer.ConsumerConfig')>, :cause=>nil}

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.