# Duplicate messages observed in Kafka when using Filebeat with disk queue

**URL:** <https://discuss.elastic.co/t/duplicate-messages-observed-in-kafka-when-using-filebeat-with-disk-queue/385693>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 30, 2026, 7:26am UTC](https://discuss.elastic.co/t/duplicate-messages-observed-in-kafka-when-using-filebeat-with-disk-queue/385693 "2026-03-30T07:26:44Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Thirupathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thirupathi/32/145431_2.png) [@Thirupathi](https://discuss.elastic.co/u/Thirupathi)\
**Post date:** [March 30, 2026, 7:26am UTC](https://discuss.elastic.co/t/duplicate-messages-observed-in-kafka-when-using-filebeat-with-disk-queue/385693/1 "2026-03-30T07:26:45Z")

</div>

We are observing duplicate messages being published to Kafka from Filebeat, even when the Kafka broker is up and stable. The issue occurs while using the disk queue (`queue.disk` ) configuration

# Kafka Output Configuration (unified for both sources)

output.kafka:  
hosts: ["xxxxxx:19092", "xxxxx:19093", "xxxx:19094"] # All three brokers set to same IP  
topic: "${KAFKA\_TOPIC}"  
key: '%{[UUID]}'  
codec.format:  
string: '%{[message]}'  
partition.hash:  
reachable\_only: true  
required\_acks: -1  
compression: gzip  
compression\_level: 4  
max\_message\_bytes: 104857600 # 100MB - supports large Apache audit logs with attachments  
version: "2.1.0"  
client\_id: "filebeat-prod"  
bulk\_max\_size: 1  
bulk\_flush\_frequency: 10ms  
channel\_buffer\_size: 256  
keep\_alive: 30s  
max\_retries: -1  
backoff.init: 1s  
backoff.max: 30s  
timeout: 2s  
broker\_timeout: 10s  
worker : 1

# Enhanced Security Configuration

#sasl.mechanism: SCRAM-SHA-512  
#username: "${KAFKA\_USERNAME}"  
#password: "${KAFKA\_PASSWORD}"  
security.protocol: PLAINTEXT  
#ssl.enabled: true  
#ssl.verification\_mode: full  
#ssl.certificate\_authorities: ["/etc/filebeat/certs/ca-cert.pem"]  
#ssl.verification\_mode: none

# Disable console logging to keep it clean

logging.to\_stderr: true # Keep enabled to see error messages during troubleshooting

# Performance and Monitoring Configuration

# Use persistent disk queue for reliable, no-duplicate, at-least-once delivery

queue.disk:  
path: /var/lib/filebeat/disk-queue  
max\_size: 20GB  
segment\_size: 100MB  
read.buffer\_size: 4MB  
write.buffer\_size: 4MB

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 30, 2026, 1:38pm UTC](https://discuss.elastic.co/t/duplicate-messages-observed-in-kafka-when-using-filebeat-with-disk-queue/385693/2 "2026-03-30T13:38:46Z")

</div>

As far as I know, this output will use at least once delivery, so duplicates are expected in some cases.

Not sure what is the issue, are all messages being duplicated? What is the percent of messages that are being duplicated?