[Solved] Logstash 5 not starting when using Kafka input version 4

Elastic Stack Logstash
1

Hallos!

I set up a logstash 5.0.0 and downgraded the kafka plugin version as described in the breaking changes document (https://www.elastic.co/guide/en/logstash/5.0/breaking-changes.html) because our kafka cluster run version 0.9
bin/logstash-plugin list --verbose | grep kafka
logstash-input-kafka (4.0.0)
logstash-output-kafka (4.0.1)
When I use this plugin, logstash fails to start, giving me the following error message:
[ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<NameError: uninitialized constant LogStash::Logger>, :backtrace=>["org/jruby/RubyModule.java:2719:in const_missing'", "/appdata/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-4.0.0/lib/logstash/inputs/kafka.rb:136:inregister'"

My start command looks like this:
logstash-5.0.0$ bin/logstash -f /path/to/configs/something.conf
and it boots for any configuration that doesn't make use of the kafka input.

Any ideas? :slight_smile: Thanks!

2

Example of what happens on a freshly downloaded copy of logstash 5.0.0:

$ cd logstash    -5.0.0
$ bin/logstash-plugin remove logstash-input-kafka; bin/logstash-plugin install --version 4.0.0 logstash-input-kafka
Removing logstash-input-kafka
Validating logstash-input-kafka-4.0.0
Installing logstash-input-kafka
Installation successful
$ bin/logstash -f ~/test_kafka_connection_logstash5.conf
Sending Logstash logs to /some/path/logstash-5.0.0/logs which is now configured via log4j2.properties.
[2016-11-07T11:04:46,075][ERROR][logstash.agent           ] Pipeline aborted due to error {:exception=>#<NameError: uninitialized constant LogStash::Logger>, :backtrace=>["org/jruby/RubyModule.java:2719:in `const_missing'", "/some/path/logstash-5.0.0/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-4.0.0/lib/logstash/inputs/kafka.rb:136:in `register'", "/some/path/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:319:in `start_inputs'", "org/jruby/RubyArray.java:1613:in `each'", "/some/path/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:318:in `start_inputs'", "/some/path/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:195:in `start_workers'", "/some/path/logstash-5.0.0/logstash-core/lib/logstash/pipeline.rb:153:in `run'", "/some/path/logstash-5.0.0/logstash-core/lib/logstash/agent.rb:250:in `start_pipeline'"]}
[2016-11-07T11:04:46,113][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2016-11-07T11:04:49,093][WARN ][logstash.agent           ] stopping pipeline {:id=>"main"}

with the config looking like this:

input
{
    kafka
    {
        bootstrap_servers => "iplist_here"
        codec => plain
        topics => ["topic_here"]
        client_id => "logstash5"
        group_id => "test"      
    }
}
filter
{

}
output
{
  stdout {}
}
3

Quick update: my question has become obsolete by the 4.1 version of the plugin which offers a patch for this.

4

I just tried installing that plugin because I was having the same error (I'm still running kafka .9), and am getting a different error now.

My install:

[root@orion-logf-1 logstash]# bin/logstash-plugin remove logstash-input-kafka
Removing logstash-input-kafka
[root@orion-logf-1 logstash]# bin/logstash-plugin install --version 4.1.0 logstash-input-kafka
Validating logstash-input-kafka-4.1.0
Installing logstash-input-kafka
Installation successful
[root@orion-logf-1 logstash]

Here's what happens now:

[2016-11-15T09:01:25,079][DEBUG][logstash.registry        ] Problems loading a plugin with {:type=>"input", :name=>#<LogStash::Registry::Plugin:0xaf3b487 @type="input", @name="kafka">, :path=>"logstash/inputs/kafka", :error=>#<LogStash::EnvironmentError: Could not find jar files under /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-4.1.0/vendor/jar-dependencies/runtime-jars/*.jar>}
[2016-11-15T09:01:25,080][DEBUG][logstash.plugin          ] Problems loading the plugin with {:type=>"input", :name=>"kafka", :path=>"logstash/inputs/kafka"}
[2016-11-15T09:01:25,086][ERROR][logstash.agent           ]  :reason=>"Couldn't find any input plugin named 'kafka'. Are you sure this is correct? Trying to load the kafka input plugin resulted in this error: Problems loading the requested plugin named kafka of type input.", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/plugin.rb:127:in `lookup'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:413:in `plugin'", "(eval):8:in `initialize'", "org/jruby/RubyKernel.java:1079:in `eval'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:90:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:195:in `create_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:87:in `register_pipeline'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:258:in `execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:174:in `run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:in `run'", "/usr/share/logstash/lib/bootstrap/environment.rb:68:in `(root)'"]}

Input config:

    kafka {
            auto_offset_reset => "smallest"
            consumer_threads => "1"
            group_id => "kafka_nginxtlsaccess"
            type => "nginx-tlsaccess"
            bootstrap_servers => "kafka.service.consul:9092"
            client_id => "orion-logf-1"
            topics => ["nginxtlsaccess"]
    }

Suggestions?

5

Use version 4.1.1 for logstash-input-kafka