# How do I know/or make sure that elastic search does not add same log file twice?

**URL:** <https://discuss.elastic.co/t/how-do-i-know-or-make-sure-that-elastic-search-does-not-add-same-log-file-twice/45736>\
**Category:** Logstash\
**Created:** [March 29, 2016, 9:54pm UTC](https://discuss.elastic.co/t/how-do-i-know-or-make-sure-that-elastic-search-does-not-add-same-log-file-twice/45736 "2016-03-29T21:54:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dani/32/8653_2.png) [@Dani](https://discuss.elastic.co/u/Dani)\
**Post date:** [March 29, 2016, 9:54pm UTC](https://discuss.elastic.co/t/how-do-i-know-or-make-sure-that-elastic-search-does-not-add-same-log-file-twice/45736/1 "2016-03-29T21:54:07Z")

</div>

When using a file input, with elasticsearch output

how do I make sure that each of the lines in the log files I generate, and have logstash process only get added once?

If I add another output, like S3 some time later, will it update that output with the older data? Or will the data need to be re-loaded from scratch and have the since\_db erased and will I need to clear the elasticsearch indexes?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [March 30, 2016, 1:58am UTC](https://discuss.elastic.co/t/how-do-i-know-or-make-sure-that-elastic-search-does-not-add-same-log-file-twice/45736/2 "2016-03-30T01:58:14Z")

</div>

I'm moving this to the Logstash area as it's more relevant there.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 30, 2016, 5:37am UTC](https://discuss.elastic.co/t/how-do-i-know-or-make-sure-that-elastic-search-does-not-add-same-log-file-twice/45736/3 "2016-03-30T05:37:52Z")

</div>

> how do I make sure that each of the lines in the log files I generate, and have logstash process only get added once?

Logstash tracks the inodes of files it has processed so unless you rotate the files by copying them to new files _and_ have `start_position => "beginning"` set you should be fine.

> If I add another output, like S3 some time later, will it update that output with the older data? Or will the data need to be re-loaded from scratch and have the since\_db erased and will I need to clear the elasticsearch indexes?

When you add additional outputs only data processed by Logstash from then on will reach the new output. So yeah, in this case you need to clear sincedb and reprocess the files.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:04am UTC](https://discuss.elastic.co/t/how-do-i-know-or-make-sure-that-elastic-search-does-not-add-same-log-file-twice/45736/4 "2017-07-06T05:04:50Z")

</div>