# How does filebeat react to files dumped at once

**URL:** <https://discuss.elastic.co/t/how-does-filebeat-react-to-files-dumped-at-once/48013>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 21, 2016, 9:50am UTC](https://discuss.elastic.co/t/how-does-filebeat-react-to-files-dumped-at-once/48013 "2016-04-21T09:50:30Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![wokmichel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wokmichel/32/48057_2.png) [@wokmichel](https://discuss.elastic.co/u/wokmichel)\
**Post date:** [April 21, 2016, 9:50am UTC](https://discuss.elastic.co/t/how-does-filebeat-react-to-files-dumped-at-once/48013/1 "2016-04-21T09:50:30Z")

</div>

Hello,  
I am trying to understand how filebeat reacts to files that are dumped at once every x hours (versus log files that are fed on the fly, line by line)

For instance, I have a file **_/log/audit.dump_** that is recreated from scracth by a script every day at midnight with new content. This is a simple csv file with a few thousands lines.

I have witnessed several behaviors from filebeat :

- The new generated file is completely read and the new lines are sent to logstash ( **expected behavior** )
- The file starts at the current offset and the first line sent to logstash is truncated
- The file is not read at all as if filebeat had not detected the file change.

Here is my filebeat prospector conf :

```
filebeat:
  prospectors:
    -
      paths:
       - /log/audit.dump

      input_type: log
      document_type: audit
      exclude_lines: ["^ACTION"]

```

Could you please shed some light on how one is supposed to approach this kind of situation ?  
Am I missing something in the way I am configuring filebeat ?

Thanks in advance for your precious help.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [April 21, 2016, 12:06pm UTC](https://discuss.elastic.co/t/how-does-filebeat-react-to-files-dumped-at-once/48013/2 "2016-04-21T12:06:02Z")

</div>

I think the reason behind the behaviour is as following:

I assume that not a new file is created but the existing file (with the same inode) is reused and the same content is put into it. This has the following consequences:

- Because the inode stays the same, filebeat assumes it is still the same file.
- If the new content is shorter then the content before, it assumes that something strange happened and starts reading from the beginning (your expected behaviour)
- If the file is longer the the previous one, it reads the lines from the old offset, which could start in the middle of a line
- If the file has the same length, nothing happens.

So the behaviour is somehow expected. To solve this issue, I recommend you to rename the old file, create the new file and dump the content, remove the old file. The reason I do not recommend delete and create directly, as this could lead to inode reuse: [https://github.com/elastic/beats/issues/1341](https://github.com/elastic/beats/issues/1341)

---

<div class="post-metadata">

**Author:** ![wokmichel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wokmichel/32/48057_2.png) [@wokmichel](https://discuss.elastic.co/u/wokmichel)\
**Post date:** [April 21, 2016, 12:21pm UTC](https://discuss.elastic.co/t/how-does-filebeat-react-to-files-dumped-at-once/48013/3 "2016-04-21T12:21:26Z")

</div>

**@ruffin** Thanks for your quick answer, I will ask the dev to try and modify the way they create the file.  
Indeed, I confirm the inode remains the same and that it matches the one fromt he registry file.  
I will keep you posted once the modification has been performed.

Christophe.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 9:52pm UTC](https://discuss.elastic.co/t/how-does-filebeat-react-to-files-dumped-at-once/48013/4 "2017-07-05T21:52:55Z")

</div>