# Elasticsearch Same document repeating multiple times with filebeat to logstash output

**URL:** <https://discuss.elastic.co/t/elasticsearch-same-document-repeating-multiple-times-with-filebeat-to-logstash-output/237833>\
**Category:** Logstash\
**Created:** [June 19, 2020, 4:30pm UTC](https://discuss.elastic.co/t/elasticsearch-same-document-repeating-multiple-times-with-filebeat-to-logstash-output/237833 "2020-06-19T16:30:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bhanu\_Praveen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhanu_praveen/32/60395_2.png) [@Bhanu\_Praveen](https://discuss.elastic.co/u/Bhanu_Praveen)\
**Post date:** [June 19, 2020, 4:30pm UTC](https://discuss.elastic.co/t/elasticsearch-same-document-repeating-multiple-times-with-filebeat-to-logstash-output/237833/1 "2020-06-19T16:30:23Z")

</div>

Hello,

Configured Filebeat to parse inputs from log file:

filebeat.inputs:

- type: log  
enabled: true  
recursive\_glob.enabled: true  
close\_inactive: 1h  
close\_removed: true  
clean\_inactive: 5h  
clean\_removed: true  
ignore\_older: 4h  
paths:
  - "/asdad/\*.json"  
json.keys\_under\_root: true  
json.add\_error\_key: true  
fields:  
index: mymetricsindex  
hc\_type: mymetrics

My json file have 10 unique rows. Elastic search was displaying correctly before from long time. Suddenly after some updates and restart, now each row is repeating 14 times and giving 140 rows in ES query.

encountered a problem with filebeat where the last file imported into elk is continuously imported, endlessly, until a new file is detected by filebeat. the new file is then continuously imported until another new file shows up, ...

Noticed in the registry (/var/lib/filebeat/registry/filebeat/data.json) that the offset for the file is never updated to the correct value, it remains 0. the filebeat logging is showing the offset correctly but does not seem to update in the registry.

Not sure why data is being repeated. For each unique row in json file, 14 documents getting created with same content. Kindly let me know if any leads for this fix?

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 17, 2020, 4:30pm UTC](https://discuss.elastic.co/t/elasticsearch-same-document-repeating-multiple-times-with-filebeat-to-logstash-output/237833/2 "2020-07-17T16:30:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.