# Filestream input sends duplicates events on restart and during operation

**URL:** <https://discuss.elastic.co/t/filestream-input-sends-duplicates-events-on-restart-and-during-operation/334951>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [June 1, 2023, 10:19am UTC](https://discuss.elastic.co/t/filestream-input-sends-duplicates-events-on-restart-and-during-operation/334951 "2023-06-01T10:19:29Z")\
**Posts on this page:** 1\
**Showing post:** 19

<div class="post-metadata">

**Author:** ![TiagoQueiroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tiagoqueiroz/32/107061_2.png) [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)\
**Post date:** [June 9, 2023, 6:02pm UTC](https://discuss.elastic.co/t/filestream-input-sends-duplicates-events-on-restart-and-during-operation/334951/19 "2023-06-09T18:02:30Z")

</div>

@michaelbu thanks a lot for all the information!

The good news is that I managed to reproduce a similar behaviour:

- Filestream input is reading any file
- The file gets truncated (`truncate --size 0 /path/to/file)
- Filestream detects the truncation and logs accordingly
- The file size is not reset in the cursor (registry)
- However, `stat` shows the correct size (zero, then it starts increasing again)

The key differences I see from my environment to yours:

- I'm running Arch Linux
- I'm using ext4 instead of xfs

Without restarting Filebeat I did not perceive any data duplication.

This is a bug in Filestream, the cursor in the registry should be updated when the file is truncated. I'll further investigate next week.

One interesting thing I noticed on your Filebeat configuration is that the input who reads Filebeat logs drops the events mentioning file truncation, here is the relevant snippet

```yaml
  processors:
  - drop_event:
      when:
        or:
          - equals:
              message: "File was truncated. Reading file from offset 0. Path=/var/log/graylog-sidecar/filebeat_stderr.log"
          - and:
            - equals:
                log.level: "info"
            - not:
                contains:
                  message: "File was truncated. Reading file from offset "

```

Did you write this Filebeat configuration or is it "auto-generated" by GrayLog?

My current theory is that GrayLog is somehow truncating the file.

However I'm very puzzled that neither `stat` nor your script is showing this.

Could you share your monitoring scrip? I would like to run some tests using the same script.

---

_[View the full topic](https://discuss.elastic.co/t/filestream-input-sends-duplicates-events-on-restart-and-during-operation/334951)._