Kafka output plugin java.lang.long error

[nich]

When I try to use the Kafka output plugin I get this variable type error:

[ERROR][logstash.outputs.kafka ] Unable to create Kafka producer from given configuration {:kafka_error_message=>org.apache.kafka.common.config.ConfigException: Invalid value 32768 for configuration receive.buffer.bytes: Expected value to be a 32-bit integer, but it was a java.lang.Long, :cause=>nil}

I am just testing, so the output configuration is very simple:

  kafka {
    codec => json
    topic_id => "logstash-test"
    bootstrap_servers => "foo.example.com:9092"
  }
}

Logstash version is 6.3.0, and Kafka output plugin version is 7.1.2. The platform is CentOS6 with Java OpenJDK version 1.8.0_171.

I have tried explicitly setting the receive_buffer_bytes parameter, with and without enclosing in quotes, but that made no difference. Does anybody have any ideas on how to fix please?

[nich]

For the benefit of anyone else with this issue, upgrading Logstash to v6.3.2 resolved this issue.

[cyb3rward0g (Roberto Rodriguez)]

FYI I am using Elastic Official Docker Images. I updated the Logstash Image 6.3.2 to 6.4.0 and the issue is the same. The kafka output plugin 7.1.2 seems to be the issue. https://github.com/logstash-plugins/logstash-output-kafka/pull/198 . FYI Version ELK 6.3.2 was working fine.

[cyb3rward0g (Roberto Rodriguez)]

Logstash 6.3.2 was released on July 24th (https://www.elastic.co/downloads/past-releases) and the Logstash Kafka Output Plugin 7.1.1 was released on June 5th (https://www.elastic.co/guide/en/logstash-versioned-plugins/current/v7.1.1-plugins-outputs-kafka.html). I am using the official Docker Image for Logstash and the 6.3.2 then must have been shipped with the Logstash Kafka Output Plugin version 7.1.1 . After updating to Docker Image 6.4.0 (released August 23rd,2018) the Logstash Kafka Output Plugin Version 7.1.2 (released August 8th, 2018) is released as part of the image. I rolled back the Kafka output plugin the following way:

bin/logstash-plugin uninstall logstash-output-kafka
bin/logstash-plugin install --version 7.1.1 logstash-output-kafka

After doing that, everything now works fine

if you want to remove version 7.1.2 and install 7.1.1 . while Logstash is starting for the firs time you can do the following:

logstash-plugin remove logstash-output-kafka
logstash-plugin install --version 7.1.1 logstash-output-kafka

you should then get the following before running logstash:

Successfully removed logstash-output-kafka
Validating logstash-output-kafka-7.1.1
Installing logstash-output-kafka
Installation successful