# Old files ignored? or indexed again with new Ids?

**URL:** <https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868>\
**Category:** Logstash\
**Created:** [November 28, 2019, 1:57pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868 "2019-11-28T13:57:58Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Incauto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/incauto/32/60149_2.png) [@Incauto](https://discuss.elastic.co/u/Incauto)\
**Post date:** [November 28, 2019, 1:57pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868/1 "2019-11-28T13:57:58Z")

</div>

I have a folder with multiple files with a name date format: "foo\_2019\_11\_23"  
I have already created indexes with those files with the folowing command

```
bin/logstash -f /etc/logstash/conf.d/foo.conf

```

now there are new files added in that folder, so I want to index those too, if I run the same command again. Will logstash create new indexes with new ids of the old files in that folder, or they will be ignored and only the new files will be added?

this is my conf file:

```
input {
    file {
            path => "/opt/processed/foo*"
            start_position => "beginning"
            sincedb_path => "/dev/null"
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 28, 2019, 2:40pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868/2 "2019-11-28T14:40:07Z")

</div>

You have told logstash, using 'sincedb\_path =\> "/dev/null"', not to persist knowledge of what it has read and what it has not read, so it will re-read everything.

---

<div class="post-metadata">

**Author:** ![Incauto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/incauto/32/60149_2.png) [@Incauto](https://discuss.elastic.co/u/Incauto)\
**Post date:** [November 28, 2019, 2:41pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868/3 "2019-11-28T14:41:26Z")

</div>

But it will create new ids for the old files?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 28, 2019, 2:43pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868/4 "2019-11-28T14:43:08Z")

</div>

If you are writing to elasticsearch, then unless you are setting document\_id based on the contents of the files, they will get new, unique, ids.

---

<div class="post-metadata">

**Author:** ![Incauto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/incauto/32/60149_2.png) [@Incauto](https://discuss.elastic.co/u/Incauto)\
**Post date:** [November 28, 2019, 3:01pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868/5 "2019-11-28T15:01:55Z")

</div>

So, my old files have 'sincedb\_path =\> "/dev/null" when they were indexed, and there is no way to avoid that logstash create new ids for my old files?

if the last is true, that means that I will have duplicates? same data, different ids?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 28, 2019, 3:24pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868/6 "2019-11-28T15:24:37Z")

</div>

> [@Incauto](#):
>
> that means that I will have duplicates? same data, different ids?

That is what I would expect.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 26, 2019, 3:24pm UTC](https://discuss.elastic.co/t/old-files-ignored-or-indexed-again-with-new-ids/209868/7 "2019-12-26T15:24:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.