# Duplication of records in ELK from Kafka

**URL:** <https://discuss.elastic.co/t/duplication-of-records-in-elk-from-kafka/93739>\
**Category:** Logstash\
**Created:** [July 19, 2017, 10:41am UTC](https://discuss.elastic.co/t/duplication-of-records-in-elk-from-kafka/93739 "2017-07-19T10:41:51Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![b2ron](https://avatars.discourse-cdn.com/v4/letter/b/e495f1/32.png) [@b2ron](https://discuss.elastic.co/u/b2ron)\
**Post date:** [July 19, 2017, 10:41am UTC](https://discuss.elastic.co/t/duplication-of-records-in-elk-from-kafka/93739/1 "2017-07-19T10:41:52Z")

</div>

CentOS release 6.9  
logstash 5.3.3-1  
java version "1.8.0\_92"  
Kafka - 2.11-0.11.0.0

I have a few hosts with logs. from these hosts filebeat logs to Kafka  
cat /etc/filebeat/filebeat.yml  
filebeat:  
prospectors:  
# container logs  
-  
paths:  
- "/mnt/log/log.log"  
json.keys\_under\_root: true  
document\_type: log  
fields:  
region: hostname  
output.kafka:  
hosts: ["kafka-host1:9092"]  
topic: "some-log"  
max\_message\_bytes: 1000000

Kafka's configuration is close to the default  
In one Kafka logs write 2-3 servers.  
Kafka's servers not integrated into the cluster  
The data stream is not large - 1-2 Mbit /s

Logstash reads these data from the kafka and transfers it to the elastic.  
Logstash reads from 4 server kafka  
kafka {  
codec =\> json  
bootstrap\_servers =\> "kafka-server-1:9092"  
client\_id =\> "kafka1"  
topics =\> ["some-log"]  
auto\_offset\_reset =\> "latest"  
}  
kafka {  
codec =\> json  
bootstrap\_servers =\> "kafka-server-2:9092"  
client\_id =\> "kafka2"  
topics =\> ["some-log"]  
auto\_offset\_reset =\> "latest"  
}  
....

But periodically there is a multiplicity duplication of records in elastic  
If I do  
[kafka-console-consumer.sh](http://kafka-console-consumer.sh) --bootstrap-server localhost:9092 --topic some-log --from-beginning  
there is a record in a single copy  
As a result of which this happens - I can not understand and where to dig, too.  
As a result, it happens - I cannot understand.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 16, 2017, 10:41am UTC](https://discuss.elastic.co/t/duplication-of-records-in-elk-from-kafka/93739/2 "2017-08-16T10:41:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.