Issue with kafka input plugin

[axelfelix (alexandre)]

Hi all,

I am on my way to upgrade all my ELK cluster. The last point is the logtsash upgrade (before 2.3, future 5.5).
Kafka version: 0.11.0.0

With logstash kafka input plugin version 2.0.6, I do not have any problem to get data from kafka and put it in elasticsearch.

But it doesn't work with logstash kafka input plugin version 5.1.8.

input plugin version 2.0.6 configuration:

input{
    kafka{
        topic_id => "test"
        group_id => "mytest"
        type => "cisco-lan"
        zk_connect => "192.168.1.1:2181"
        codec => plain
    }
}

input plugin version 5.1.8 configuration:

input{
    kafka{
        topics => ["test"]
        group_id => "mytest"
        type => "cisco-lan"
        bootstrap_servers => "192.168.1.1:2181"
        codec => plain
    }
}

I already try to change in server.properties configuration the line "advertised.host.name". No more success.

When I take a tcpdum I get Reset connection, I don't know why...

If you have any idea...

Thanks in advance,
Alex

[axelfelix (alexandre)]

Solved using the port 9092 instead the port 2181.

Somebody knows why do I have to change the port between version 2.3 and 5.5 of Logstash ?

[mwysokin (Michal Wysokinski)]

@axelfelix

Somebody

Yes, now you are connecting directly to kafka using so called "new consumer API". In the previous version you were actually connecting to ZooKeeper and not kafka. I assume you have them running on the same machine. That's why you didn't have to change the IP address.