# Filebeat: Kafka Input doesn't push the topic offset
**URL:** https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436
**Category:** Beats
**Tags:** filebeat
**Created:** [December 20, 2021, 7:47am UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436 "2021-12-20T07:47:01Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Giero](https://discuss.elastic.co/u/Giero)
#### Post date: [December 20, 2021, 7:47am UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436/1 "2021-12-20T07:47:01Z")
I'm using Filebeat 7.16.0 with Kafka Input and Elasticsearch Output.
Previously I had java application to take care of this flow.
I'm using the same consumer group as previous java application and messages appear in Elasticsearch, but the topic offset stays the same. It's an issue, because it's on Kubernetes and if the pod will be restarted, the filebeat will duplicate the messages which it previously processed.
Current state of topic:
```auto
PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG
11 5986 6399 413
4 6090 6497 407
7 6195 6551 356
1 6045 6468 423
5 6177 6593 416
0 6074 6485 411
9 5977 6414 437
8 6194 6562 368
2 5991 6397 406
6 6160 6558 398
10 6055 6466 411
3 6028 6461 433
```
filebeat.yml:
```auto
filebeat.shutdown_timeout: 300s
http.enabled: true
logging:
level: warn
filebeat.inputs:
- type: kafka
hosts: kafka-host:19090
topics: kafka-topic
group_id: kafka-consumer-group
sasl.mechanism: PLAIN
username: username
password: password
output.elasticsearch:
hosts: https://elastic-host:443
username: username
password: password
bulk_max_size: 25
compression_level: 9
index: index
processors:
- decode_json_fields:
fields: message
target: ""
overwrite_keys: true
setup.ilm.enabled: true
setup.ilm.policy_name: index
setup.ilm.rollover_alias: index
setup.ilm.policy_file: /etc/ilm-policy.json
setup.ilm.overwrite: true
setup.template.name: index
setup.template.pattern: index-*
setup.template.fields: /etc/fields.yml
setup.template.overwrite: true
setup.template.append_fields:
- name: "@timestamp"
type: date
```
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [December 20, 2021, 1:55pm UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436/2 "2021-12-20T13:55:38Z")
I think there is an option to select the direction ( `initial_offset`: `oldest`, `newest`).
---
### Author:  [@Giero](https://discuss.elastic.co/u/Giero)
#### Post date: [December 20, 2021, 2:41pm UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436/3 "2021-12-20T14:41:53Z")
Cześć,
Thank you for your comment!
It didn't seem to change anything though, the offset is still frozen.
---
### Author:  [@Giero](https://discuss.elastic.co/u/Giero)
#### Post date: [December 22, 2021, 6:27am UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436/4 "2021-12-22T06:27:12Z")
Any ideas? Should I create a bug for this?
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [December 24, 2021, 3:18pm UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436/5 "2021-12-24T15:18:41Z")
Could you please post more details regarding the Kafka instance? Version, with/without ZooKeeper, any debug logs.
We can open an issue for Beats if only we can confirm that this is a bug indeed.
---
### Author:  [@Giero](https://discuss.elastic.co/u/Giero)
#### Post date: [January 11, 2022, 1:02pm UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436/6 "2022-01-11T13:02:40Z")
Hi, so I wanted to make easy steps to reproduce on minikube:
> **[GitHub - Gier32o/filebeat-issue](https://github.com/Gier32o/filebeat-issue)**
>
> Contribute to Gier32o/filebeat-issue development by creating an account on GitHub.
---
### Author:  [@Giero](https://discuss.elastic.co/u/Giero)
#### Post date: [January 24, 2022, 12:54pm UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436/7 "2022-01-24T12:54:07Z")
Will be creating a bug in few days
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 21, 2022, 2:54pm UTC](https://discuss.elastic.co/t/filebeat-kafka-input-doesnt-push-the-topic-offset/292436/8 "2022-02-21T14:54:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.