Logstash unable to read messages from kafka

Elastic Stack Logstash
1

Following is the configuration :

input {
  kafka {
    codec => "json"
    bootstrap_servers => "10.0.20.50:4000,10.0.20.50:4001,10.0.20.4:4000,10.0.20.4:4001,10.0.20.62:4000,10.0.20.62:4001"
    group_id => "logstash"
    topics => "logstash_logs"
    consumer_threads => 1
  }
}
output {
  stdout { codec => rubydebug { metadata => true}}
}

Same configuration is working with 2.x , after upgrading to 5.4 and installing logstash input plugin (4.2.0) version. Logstash is unable to read messages from kafka. We have migrated whole stack to 5.x only logstash is remaining.

Logs are at following
https://github.com/rsingh2411/test/blob/master/logstash-plain.log

I tried latest 5.6 version, following are the logs

from the logs it shows that logstash is able to connect to first broker but, its not able to connect to coordinator, then it tries to connect to other cooridinator server

Discovered coordinator thor12-worker-4:4000 (id: 2147483646 rack: null) for group logstash.
[2017-10-04T10:50:45,104][DEBUG][org.apache.kafka.clients.NetworkClient] Initiating connection to node 2147483646 at thor12-worker-4:4000.
[2017-10-04T10:50:49,678][DEBUG][logstash.pipeline        ] Pushing flush onto pipeline
[2017-10-04T10:50:50,115][DEBUG][org.apache.kafka.clients.NetworkClient] Error connecting to node 2147483646 at thor12-worker-4:4000:
java.io.IOException: Can't resolve address: thor12-worker-4:4000
[2017-10-04T10:50:50,120][INFO ][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] Marking the coordinator thor12-worker-4:4000 (id: 2147483646 rack: null) dead for group logstash
[2017-10-04T10:50:50,120][DEBUG][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator] Cannot auto-commit offsets for group logstash since the coordinator is unknown
[2017-10-04T10:50:50,120][DEBUG][org.apache.kafka.clients.consumer.internals.AbstractCoordinator] Sending coordinator request for group logstash to broker thor12-worker-1:4000 (id: 3 rack: null)
[2017-10-04T10:50:50,121][DEBUG][org.apache.kafka.clients.NetworkClient] Initiating connection to node 3 at thor12-worker-1:4000.
2

Issue resolved. using alpine 3.3 base image , hence was unable to resolve dns.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.