# Note in kafka output

**URL:** <https://discuss.elastic.co/t/note-in-kafka-output/299336>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 10, 2022, 1:03pm UTC](https://discuss.elastic.co/t/note-in-kafka-output/299336 "2022-03-10T13:03:01Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![YuWatanabe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yuwatanabe/32/13259_2.png) [@YuWatanabe](https://discuss.elastic.co/u/YuWatanabe)\
**Post date:** [March 10, 2022, 1:03pm UTC](https://discuss.elastic.co/t/note-in-kafka-output/299336/1 "2022-03-10T13:03:01Z")

</div>

Hello.

The [doc](https://www.elastic.co/guide/en/beats/filebeat/7.17/kafka-output.html#kafka-output) notes _record_ in kafka topic will be deleted _immediately_ .

> it’s going to be immediately discarded since the timestamp value is before the last 7 days.

However, kafka is deleting the segment according to [log.retention.check.interval.ms](https://kafka.apache.org/27/documentation.html#brokerconfigs_log.retention.check.interval.ms) . Thus I thought records are not deleted immediately.

I have verified this comparing the timestamp of _filebeat_ output and _kafka-console-consumer.sh_ . By immediately consuming after topic is created and checked that consumer was able to consume messages.

```auto
# Filebeat
2022-03-10T11:33:36.159Z INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to kafka(b-1.hoge.uui4gf.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-2.hoge.uui4gf.c2.kafka.ap-northeast-1.amazonaws.com:9092,b-3.hoge.uui4gf.c2.kafka.ap-northeast-1.amazonaws.com:9092) established

# kafka-console-consumer
[ssm-user@ip-10-0-10-48 kafka_2.13-2.7.0]$ bin/kafka-console-consumer.sh --bootstrap-server b-1.hoge.uui4gf.c2.kafka.ap-northeast-1.amazonaws.com:9092 --topic i-02abf41b2811cc402-google-workspace --from-beginning --property print.timestamp=true &> messages.log
ssm-user@ip-10-0-10-48 kafka_2.13-2.7.0]$ ls -l
total 2972
-rw-r--r-- 1 ssm-user ssm-user 29975 Dec 16 2020 LICENSE
-rw-r--r-- 1 ssm-user ssm-user 337 Dec 16 2020 NOTICE
drwxr-xr-x 3 ssm-user ssm-user 4096 Dec 16 2020 bin
drwxr-xr-x 2 ssm-user ssm-user 4096 Dec 16 2020 config
drwxr-xr-x 2 ssm-user ssm-user 8192 Mar 3 01:35 libs
-rw-rw-r-- 1 ssm-user ssm-user 2984784 Mar 10 11:34 messages.log
drwxr-xr-x 2 ssm-user ssm-user 44 Dec 16 2020 site-docs

```

With broker , I have used AWS MSK kafka ver `2.7.0` only changing below broker settings.

```auto
  auto.create.topics.enable = true
  delete.topic.enable = true
  num.partitions = 1
  default.replication.factor = 3

```

So I thought something like ,

> depending on [log.retention.hours](https://kafka.apache.org/documentation/#brokerconfigs_log.retention.hours) setting (default 7days), it will be discarded immediately after [the log cleaner](https://kafka.apache.org/documentation/#design_compactionconfig) takes in place.

May I ask for review with my thought ?

Best Regards,  
Yu Watanabe

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 7, 2022, 3:03pm UTC](https://discuss.elastic.co/t/note-in-kafka-output/299336/2 "2022-04-07T15:03:31Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.