How to re fetch data from Kafka, from the beginning of topic, without changing consumer group? (resetting consumer group offset)

Elastic Stack Logstash
1

Hi,

I am using ELK GA 5.0.0. I have a logstash instance to pull data from a kafka topic, like below;

kafka{
	group_id => "kafkagroupA"
	topics => ["kafkatopicA"]
	bootstrap_servers => "192.168.0.1:9091,192.168.0.2:9093,192.168.0.3:9092"
	codec => json
}

due to some config errors, my logs of past 2 days hasn't entered elasticsearch, but logstash was consuming. Now, I have corrected the config, and logs are coming fine. How can I get the old logs (past 2 days)? It is there in Kafka topic. I cant change the consumer group (group_id). Is there any way to achieve this?

Thanks in advance.

2

The offsets for the Consumer Group are stored in Kafka itself per partition.

You need to use Kafka tools to do this. Here is a link to an article that may help - https://community.hortonworks.com/articles/81357/manually-resetting-offset-for-a-kafka-topic.html

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.