# Filebeat/Logstash and duplicate indexes

**URL:** <https://discuss.elastic.co/t/filebeat-logstash-and-duplicate-indexes/149334>\
**Category:** Elasticsearch\
**Created:** [September 20, 2018, 5:53pm UTC](https://discuss.elastic.co/t/filebeat-logstash-and-duplicate-indexes/149334 "2018-09-20T17:53:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![0a118576f418da54acef](https://avatars.discourse-cdn.com/v4/letter/0/8491ac/32.png) [@0a118576f418da54acef](https://discuss.elastic.co/u/0a118576f418da54acef)\
**Post date:** [September 20, 2018, 5:53pm UTC](https://discuss.elastic.co/t/filebeat-logstash-and-duplicate-indexes/149334/1 "2018-09-20T17:53:24Z")

</div>

Hi all

Question about Log stash and filebeat , at my work I sync or copy Apache log file from amazon S3 bucket (but the issue is the same if i add data to the local file manually ), this step is with filebeat that transfer the apache log data to Logstash (and from logstash to elastic) , the first time that i transfer the file from filebeat to logstash it's working great ,but when i add data to the file by manually add lines or by sync form s3 the indexes duplicate itself , meaning lets say that in kibana i see 100,000 (lines) ,and only add few lines 10 for example to the log file i will see 200010 instead of 100010.

help please 🙂  
thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 22, 2018, 7:02pm UTC](https://discuss.elastic.co/t/filebeat-logstash-and-duplicate-indexes/149334/2 "2018-09-22T19:02:52Z")

</div>

Logstash and Filebeat keep track of files through their node and assume data will be appended to them. If you copy over a file from S3 or manually add lines through an editor, a new file (as node will change) with the same name is created, and Logstash and Filebeat will see this as a new file and process it entirely.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 20, 2018, 7:03pm UTC](https://discuss.elastic.co/t/filebeat-logstash-and-duplicate-indexes/149334/3 "2018-10-20T19:03:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.