# How does Logstash behave when Input file content is deleted?

**URL:** <https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570>\
**Category:** Logstash\
**Created:** [October 19, 2017, 2:19pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570 "2017-10-19T14:19:26Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [October 19, 2017, 2:19pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/1 "2017-10-19T14:19:26Z")

</div>

Hi,

I have a Logstash setup where it reads file(Linux server logs) as an input. But for every 5 days, content in that file will be deleted and new logs will be placed in the input file. Will Logstash be reading lines from beginning? I am confused here. Can someone please explain?

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 19, 2017, 3:41pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/2 "2017-10-19T15:41:09Z")

</div>

_Exactly_ what happens to the file? Is it file deleted? Or truncated? Or something else?

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [October 20, 2017, 3:37pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/3 "2017-10-20T15:37:59Z")

</div>

Sorry for the confusion. File name is server.log and for every 5 days it becomes server-x.log and a new file is created with the name server.log. Process repeats for every 5 days.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 23, 2017, 6:06am UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/4 "2017-10-23T06:06:30Z")

</div>

See [https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#\_file\_rotation\_2](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_file_rotation_2).

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [October 24, 2017, 3:38pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/5 "2017-10-24T15:38:30Z")

</div>

I've read that, but I am not clear at that. Can you please explain taking my current scenario as an example?  
And this is how my input config looks like,

```
input{

        file{
                path=>"/opt/logs/server.log"
                start_position=> "beginning"
                sincedb_path => "/opt/elk/logstash/sincedb/success_sincedb.log"
                type=>"success"

                codec => multiline {
                        pattern => "^success"
                        negate => true
                        what => previous
                        #auto_flush_interval => 10
                        max_lines => 100000
                        max_bytes => "1000 MiB"
                }
        }
}
```

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [October 26, 2017, 11:02am UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/6 "2017-10-26T11:02:24Z")

</div>

Hi Magnus, can you please explain?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 26, 2017, 11:06am UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/7 "2017-10-26T11:06:00Z")

</div>

You're not saying what part is unclear so I'm not sure what to explain. When the file is renamed Logstash will discover this and start reading the newly created file instead.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [October 26, 2017, 3:16pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/8 "2017-10-26T15:16:17Z")

</div>

> [@magnusbaeck](#):
>
> When the file is renamed Logstash will discover this and start reading the newly created file instead.

When it starts reading the newly created file(since in the input I declared "path=\>"/opt/logs/server.log""), how would sincedb file change?

The reason I am asking this is - when it's reading server.log, Logstash stores in the sincedb file that it has read till 100th line and on the fifth day when server.log is renamed to server1.log and a new file server.log is created(with the same name), how will Logstash behave here? Will it read from 1st line or from 101 line?

Or is it like sincedb file gets updated only when Logstash is shut down? If that's the case then what happens when Logstash shutting down and renaming the file happens at the same time?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 26, 2017, 3:19pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/9 "2017-10-26T15:19:40Z")

</div>

> how would sincedb file change?

A new sincedb entry will be created since the new file has a new inode number. Logstash tracks file by inode numbers, not names. See the documentation.

> The reason I am asking this is - when it's reading server.log, Logstash stores in the sincedb file that it has read till 100th line

It stores a byte offset, not a line number. Again, see the documentation.

> and on the fifth day when server.log is renamed to server1.log and a new file server.log is created(with the same name), how will Logstash behave here? Will it read from 1st line or from 101 line?

The file will be read from the top.

---

<div class="post-metadata">

**Author:** ![amruth](https://avatars.discourse-cdn.com/v4/letter/a/43a26b/32.png) [@amruth](https://discuss.elastic.co/u/amruth)\
**Post date:** [October 26, 2017, 4:19pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/10 "2017-10-26T16:19:53Z")

</div>

Thanks a lot for the detailed explanation.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 23, 2017, 4:20pm UTC](https://discuss.elastic.co/t/how-does-logstash-behave-when-input-file-content-is-deleted/104570/11 "2017-11-23T16:20:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
