Hi,
I have a set-up with Kafka and ELK. In the logstash side, I frequently receive the following warning message.
23:41:29.767 [Ruby-0-Thread-7: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.0/lib/logstash/inputs/kafka.rb:225] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Marking the coordinator x.x.x.x:x (id: 2147483646 rack: null) dead for group logstash
23:41:29.770 [Ruby-0-Thread-7: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.0/lib/logstash/inputs/kafka.rb:225] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Discovered coordinator x.x.x.x:x (id: 2147483646 rack: null) for group logstash.
23:41:29.896 [Ruby-0-Thread-7: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.0/lib/logstash/inputs/kafka.rb:225] WARN org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Auto offset commit failed for group logstash: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured session.timeout.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records.
...
23:41:29.990 [Ruby-0-Thread-7: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.0/lib/logstash/inputs/kafka.rb:225] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Revoking previously assigned partitions [test-0] for group logstash
23:41:29.990 [Ruby-0-Thread-7: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.0/lib/logstash/inputs/kafka.rb:225] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining group logstash
23:41:29.993 [Ruby-0-Thread-7: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.0/lib/logstash/inputs/kafka.rb:225] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - (Re-)joining group logstash
23:41:33.300 [Ruby-0-Thread-7: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.0/lib/logstash/inputs/kafka.rb:225] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator - Successfully joined group logstash with generation 647
23:41:33.300 [Ruby-0-Thread-7: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.0/lib/logstash/inputs/kafka.rb:225] INFO org.apache.kafka.clients.consumer.internals.ConsumerCoordinator - Setting newly assigned partitions [test-0] for group logstash
For the first two lines, I guess that kafka considers my logstash client does not respond in a time period, so it disconnects. But reconnect again. For the 3rd line, it is a batch of same warning message. I have followed it, setting session_timeout_ms to "25000" and set max_poll_records to "1". But I still got same above warning message pattern in logstash. Is there any setting I missed? Thanks.
My logstash input
input {
kafka {
topics => ["test"]
bootstrap_servers => "${KAFKA_SERVERS}"
type => 'kafka-input'
max_poll_records => '1'
session_timeout_ms => "25000"
}
}
Michael