with Logstash 5.2, I'm testing sending data to kafka (using Kerberos). Below is the kakfa confiugration:
kafka {
bootstrap_servers => "xxx"
jaas_path => "/usr/hdp/current/kafka-broker/config/kafka_client_jaas.conf"
kerberos_config => "/etc/krb5.conf"
security_protocol => "SASL_PLAINTEXT"
sasl_kerberos_service_name => "kafka"
topic_id => "test"
}
I'm sure that the bootstrap_servers and jass_path configurations are correct, as I use the same setting for KafkaConsoleProducer to send data successfully. Logstash failed to startup, giving below exception.
[2017-02-23T10:24:41,255][ERROR][logstash.outputs.kafka ] Unable to create Kafka producer from given configuration {:kafka_error_message=>org.apache.kafka.common.KafkaException: Failed to construct kafka producer}
[2017-02-23T10:24:41,263][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>org.apache.kafka.common.KafkaException: Failed to construct kafka producer, :backtrace=>["org.apache.kafka.clients.producer.KafkaProducer.(org/apache/kafka/clients/producer/Ka
fkaProducer.java:335)", "org.apache.kafka.clients.producer.KafkaProducer.(org/apache/kafka/clients/producer/KafkaProducer.java:188)", "java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:423)", "RUBY.create_producer(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.3/lib/logstash/outputs/kafka.rb:235)", "RUBY.register(/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-kafka-5.1.3/lib/logstash/outputs/kafka.rb:171)", "RUBY.register(/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8)", "RUBY.register(/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:37)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:234)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1613)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:234)", "RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:188)", "RUBY.start_pipeline(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:302)", "java.lang.Thread.run(java/lang/Thread.java:745)"]}
I use command line to start the logstash
/usr/share/logstash/bin/logstash --path.settings /etc/logstash --log.level trace -f /etc/logstash/conf.d/shipper_kafka_elk.conf
I'm not sure what goes wrong, please help!