Issue with kafka input plugin

Hi all,

I am on my way to upgrade all my ELK cluster. The last point is the logtsash upgrade (before 2.3, future 5.5).
Kafka version: 0.11.0.0

With logstash kafka input plugin version 2.0.6, I do not have any problem to get data from kafka and put it in elasticsearch.

But it doesn't work with logstash kafka input plugin version 5.1.8.

input plugin version 2.0.6 configuration:

input{
    kafka{
        topic_id => "test"
        group_id => "mytest"
        type => "cisco-lan"
        zk_connect => "192.168.1.1:2181"
        codec => plain
    }
}

input plugin version 5.1.8 configuration:

input{
    kafka{
        topics => ["test"]
        group_id => "mytest"
        type => "cisco-lan"
        bootstrap_servers => "192.168.1.1:2181"
        codec => plain
    }
}

I already try to change in server.properties configuration the line "advertised.host.name". No more success.

When I take a tcpdum I get Reset connection, I don't know why...

If you have any idea...

Thanks in advance,
Alex

Solved using the port 9092 instead the port 2181.

Somebody knows why do I have to change the port between version 2.3 and 5.5 of Logstash ?

Yes, now you are connecting directly to kafka using so called "new consumer API". In the previous version you were actually connecting to ZooKeeper and not kafka. I assume you have them running on the same machine. That's why you didn't have to change the IP address.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.