# Under special log rolling strategy, filebeat repeatedly collects logs

**URL:** <https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 11, 2020, 2:36am UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036 "2020-03-11T02:36:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![xiongjunkun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xiongjunkun/32/64189_2.png) [@xiongjunkun](https://discuss.elastic.co/u/xiongjunkun)\
**Post date:** [March 11, 2020, 2:36am UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036/1 "2020-03-11T02:36:51Z")

</div>

I have a special log rolling strategy. The application keeps writing logs to file f. When file f reaches the rolling threshold, f is renamed to f1, the original f1 is renamed to f2, and so on. Recently, we found There are a lot of duplicates in the logs collected by filebeat, which is suspected to be caused by this log rolling strategy, but there is no specific reason for the duplicate logs. Does anyone know the cause of the duplicate logs?

---

<div class="post-metadata">

**Author:** ![mtojek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mtojek/32/63863_2.png) [@mtojek](https://discuss.elastic.co/u/mtojek)\
**Post date:** [March 11, 2020, 8:17am UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036/2 "2020-03-11T08:17:27Z")

</div>

Could you please explain where do you see duplicates? Did check their size and content too? "ls -la" ?

---

<div class="post-metadata">

**Author:** ![xiongjunkun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xiongjunkun/32/64189_2.png) [@xiongjunkun](https://discuss.elastic.co/u/xiongjunkun)\
**Post date:** [March 18, 2020, 2:24pm UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036/3 "2020-03-18T14:24:59Z")

</div>

Sorry for not responding in time. My program writes 5 pieces of data in a file, and then performs log scrolling. What I have seen so far is: When filebeat repeatedly collects data, it always collects all data of the file repeatedly Instead of several pieces of data.

---

<div class="post-metadata">

**Author:** ![xiongjunkun](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xiongjunkun/32/64189_2.png) [@xiongjunkun](https://discuss.elastic.co/u/xiongjunkun)\
**Post date:** [March 18, 2020, 2:29pm UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036/4 "2020-03-18T14:29:30Z")

</div>

After several tests, I found that the file has inode reuse. But after a series of analysis, I found that it is not the inode reuse that caused the filebeat to repeatedly collect logs. In addition, my configuration is a basic File, and I wrote the log collected by filebeat to another file.I analyzed this file and found that filebeat has repeatedly collected.

```auto
filebeat.inputs:
- type: log
  paths:
    - /home/xxx/log/*.log

output.file:
  path: "/home/xiongjunkun/filebeat"
  filename: filebeat
  rotate_every_kb: 1073741824
  number_of_files: 16
  permissions: 0600

logging.level: debug
logging.selectors: []
logging.to_stderr: false
logging.to_syslog: false
logging.to_eventlog: false
logging.to_files: true
logging.files:
  path: /home/xxx/filebeat_log/
  name: filebeat_log.log
  rotateonstartup: true
  rotateeverybytes: 104857600 
  keepfiles: 64
  permissions: 0600

```

---

<div class="post-metadata">

**Author:** ![hailin0](https://avatars.discourse-cdn.com/v4/letter/h/5daacb/32.png) [@hailin0](https://discuss.elastic.co/u/hailin0)\
**Post date:** [April 10, 2020, 3:03am UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036/5 "2020-04-10T03:03:43Z")

</div>

I encountered the same problem

---

<div class="post-metadata">

**Author:** ![hailin0](https://avatars.discourse-cdn.com/v4/letter/h/5daacb/32.png) [@hailin0](https://discuss.elastic.co/u/hailin0)\
**Post date:** [April 10, 2020, 3:05am UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036/6 "2020-04-10T03:05:18Z")

</div>

Has your problem been solved?

---

<div class="post-metadata">

**Author:** ![hailin0](https://avatars.discourse-cdn.com/v4/letter/h/5daacb/32.png) [@hailin0](https://discuss.elastic.co/u/hailin0)\
**Post date:** [April 10, 2020, 3:38am UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036/7 "2020-04-10T03:38:21Z")

</div>

hi

Are there any new discoveries?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2020, 3:38am UTC](https://discuss.elastic.co/t/under-special-log-rolling-strategy-filebeat-repeatedly-collects-logs/223036/8 "2020-05-08T03:38:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.