How to re fetch data from Kafka, from the beginning of topic, without changing consumer group? (resetting consumer group offset)



I am using ELK GA 5.0.0. I have a logstash instance to pull data from a kafka topic, like below;

	group_id => "kafkagroupA"
	topics => ["kafkatopicA"]
	bootstrap_servers => ",,"
	codec => json

due to some config errors, my logs of past 2 days hasn't entered elasticsearch, but logstash was consuming. Now, I have corrected the config, and logs are coming fine. How can I get the old logs (past 2 days)? It is there in Kafka topic. I cant change the consumer group (group_id). Is there any way to achieve this?

Thanks in advance.

(Guy Boertje) #2

The offsets for the Consumer Group are stored in Kafka itself per partition.

You need to use Kafka tools to do this. Here is a link to an article that may help -

(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.