I have a Kafka cluster that I am reading data from. However I am seeing offsets being committed to the cluster much more frequently than what I have in my settings. Additionally I am seeing offsets being written at very sporadic intervals. I have seen commits as frequently as 1ms apart for the same partition/thread. This appears to be causing issues on my Kafka cluster. Does anyone have an idea what could be causing this?
Kafka broker version - 0.10.0.1
Logstash version - 5.6.0
Kafka input plugin version - 5.1.11
Logstash input config:
input {
kafka {
bootstrap_servers => '10.39.108.221:9092,10.39.108.219:9092,10.39.108.223:9092'
topics_pattern => "if_.*?"
auto_offset_reset => "earliest"
value_deserializer_class => "org.apache.kafka.common.serialization.ByteArrayDeserializer"
client_id => "logstash_output_102"
consumer_threads => 2
group_id => "logstash_testing_02"
auto_commit_interval_ms => "20000"
}
}
Logstash logs - last 100 line, for one partition:
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250059295,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575194550 for partition if_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250062372,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575221787 for partition .if_stat-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250066401,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575262141 for partition if_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250067072,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575270230 for partition if_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250067825,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575278114 for partition if_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250070424,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575298156 for partition if_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250071979,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575301655 for partition if_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250079151,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575322309 for partition if_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250079160,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575322309 for partition if_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250079295,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp_prod_logstash_01 committed offset 3575322998 for partition if_stats-0"}}