# Kafka input - offset committing to frequently
**URL:** https://discuss.elastic.co/t/kafka-input-offset-committing-to-frequently/104262
**Category:** Logstash
**Created:** [October 17, 2017, 2:29pm UTC](https://discuss.elastic.co/t/kafka-input-offset-committing-to-frequently/104262 "2017-10-17T14:29:27Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@blacklobo](https://discuss.elastic.co/u/blacklobo)
#### Post date: [October 17, 2017, 2:29pm UTC](https://discuss.elastic.co/t/kafka-input-offset-committing-to-frequently/104262/1 "2017-10-17T14:29:27Z")
I have a Kafka cluster that I am reading data from. However I am seeing offsets being committed to the cluster much more frequently than what I have in my settings. Additionally I am seeing offsets being written at very sporadic intervals. I have seen commits as frequently as 1ms apart for the same partition/thread. This appears to be causing issues on my Kafka cluster. Does anyone have an idea what could be causing this?
Kafka broker version - 0.10.0.1
Logstash version - 5.6.0
Kafka input plugin version - 5.1.11
Logstash input config:
input {
kafka {
bootstrap\_servers =\> '10.39.108.221:9092,10.39.108.219:9092,10.39.108.223:9092'
topics\_pattern =\> "if\_.\*?"
auto\_offset\_reset =\> "earliest"
value\_deserializer\_class =\> "org.apache.kafka.common.serialization.ByteArrayDeserializer"
client\_id =\> "logstash\_output\_102"
consumer\_threads =\> 2
group\_id =\> "logstash\_testing\_02"
auto\_commit\_interval\_ms =\> "20000"
}
}
Logstash logs - last 100 line, for one partition:
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250059295,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575194550 for partition if\_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250062372,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575221787 for partition .if\_stat-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250066401,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575262141 for partition if\_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250067072,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575270230 for partition if\_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250067825,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575278114 for partition if\_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250070424,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575298156 for partition if\_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250071979,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575301655 for partition if\_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250079151,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575322309 for partition if\_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250079160,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575322309 for partition if\_stats-0"}}
{"level":"DEBUG","loggerName":"org.apache.kafka.clients.consumer.internals.ConsumerCoordinator","timeMillis":1508250079295,"thread":"Ruby-0-Thread-37: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-kafka-5.1.11/lib/logstash/inputs/kafka.rb:229","logEvent":{"message":"Group sp\_prod\_logstash\_01 committed offset 3575322998 for partition if\_stats-0"}}
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 14, 2017, 2:29pm UTC](https://discuss.elastic.co/t/kafka-input-offset-committing-to-frequently/104262/2 "2017-11-14T14:29:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.