# Filebeat will output dumplication when logrotate happened **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [September 18, 2016, 6:51am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799 "2016-09-18T06:51:42Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![haha](https://avatars.discourse-cdn.com/v4/letter/h/59ef9b/32.png) [@haha](https://discuss.elastic.co/u/haha)\ **Post date:** [September 18, 2016, 6:51am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/1 "2016-09-18T06:51:42Z")
Hi, when I use filebeat to collect the log data. I found that the output file have many dumplications of a log entry. The log data is generated by a python script. Every log entry has a unique number and every 300 entries will trigger a logrotate. First, I start the filebeat, then run the script to generate the log. At last, I found that the 1-300 entries occuer once, 300-600 entries occure twice, 600-900 entries occure three times and so on. following is my test data: python script: [https://github.com/shanmaoye/filebeat-test/blob/master/logrotate\_test.py](https://github.com/shanmaoye/filebeat-test/blob/master/logrotate_test.py) filebeat debug level log: [https://raw.githubusercontent.com/shanmaoye/filebeat-test/master/beat.log](https://raw.githubusercontent.com/shanmaoye/filebeat-test/master/beat.log) the output of the filebeat: [https://raw.githubusercontent.com/shanmaoye/filebeat-test/master/filebeat](https://raw.githubusercontent.com/shanmaoye/filebeat-test/master/filebeat) filebeat configuation: ```auto filebeat: prospectors: - paths: - /var/log/shanmao/shanmao.log - /var/log/shanmao/shanmao.log.1 input_type: log exclude_lines: ["^$"] output: file: path: "/tmp/filebeat" filename: filebeat rotate_every_kb: 100000 number_of_files: 7 shipper: logging: level: debug to_files: true files: path: /var/log/mybeat name: beat.log rotateeverybytes: 10485760 ``` logrotate configuration: /var/log/shanmao/shanmao.log { rotate 40 size 15k missingok notifempty compress delaycompress } ```auto my system info: ubuntu_14.04_64bit, version of the filebeat is 1.2.3 I wonder why does this happen? Is my configuration wrong? Thanks. ``` ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 19, 2016, 7:56am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/2 "2016-09-19T07:56:09Z")
Could you check fi you see the same behaviour with the most recent snapshot? [https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/](https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/) It can happen in the 1.x release that under heavy load and heavy file rotation some race conditions occur. ---
**Author:** ![nan008](https://avatars.discourse-cdn.com/v4/letter/n/e9bcb4/32.png) [@nan008](https://discuss.elastic.co/u/nan008)\ **Post date:** [September 19, 2016, 1:13pm UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/3 "2016-09-19T13:13:44Z")
Hi, Just to be accurate - filebeat should not duplicate entries after log rotation happens? We are starting with ELK, have everything set up and sending the logs to ES and at the moment we are trying to log rotate every week. We are using Centos 7 logrotate with similar conf to OP. The only worrying part is if we rotate log and then tell filebeat to track both logs for 5 minutes to not loose any entries, will filebeat treat the rotate file production.log1 as a new log and will resend all the data to ES? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [September 19, 2016, 1:26pm UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/4 "2016-09-19T13:26:37Z")
filebeat uses inodes to keep track if file identity. It will detect `production.log1` being a renamed file (if inode didn't change, check with `ls` command) and will continue processing where it left of in case anything was missing. The newly created `production.log` should be processed from the very beginning. ---
**Author:** ![haha](https://avatars.discourse-cdn.com/v4/letter/h/59ef9b/32.png) [@haha](https://discuss.elastic.co/u/haha)\ **Post date:** [September 20, 2016, 2:06am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/5 "2016-09-20T02:06:27Z")
I use filebeat-6.0.0-alpha1-SNAPSHOT-linux-x86\_64.tar.gz to check as the above. But, I still see the same behaviour. ---
**Author:** ![haha](https://avatars.discourse-cdn.com/v4/letter/h/59ef9b/32.png) [@haha](https://discuss.elastic.co/u/haha)\ **Post date:** [September 20, 2016, 2:19am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/6 "2016-09-20T02:19:13Z")
I think it will not resend the production.log.1 as a new log if you don't add it to the configuration of the filebeat. Because you can see the same source of the dumplication in the beat.log. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 20, 2016, 6:48am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/7 "2016-09-20T06:48:39Z")
As Steffen wrote, filebeat does track files by inode and not name. So if a file is rotated, data will not be resent as it still the same file. @haha I'm still trying to figure out what goes wrong in your case especially that you mention you also see this under the 5.0 alpha snapshot. Any chance to post the debug log of the 5.0 run? How do you exactly start filebeat? ---
**Author:** ![haha](https://avatars.discourse-cdn.com/v4/letter/h/59ef9b/32.png) [@haha](https://discuss.elastic.co/u/haha)\ **Post date:** [September 20, 2016, 11:37am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/8 "2016-09-20T11:37:39Z")
I'm sorry. I think my python script cause this problem. I use python logging module to write the log. At first, the filebeat get the right output with the logrotate. But the script write all log in the production.log.1. That, I think is a wrong way to use logging and logrotate. So , I change my script to close filehandler before logrotate and open a new filehandler after logrotate and get the above problem. Today, I try to use copytruncate in the logrotate and don't refresh the filehandler when logrotate. Then, I get the correct filebeat output. So, I think my way to use logrotate and logging may not be suitable. Thank you for you help. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [September 20, 2016, 11:52am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/9 "2016-09-20T11:52:26Z")
We use Python for some of our system tests and use the internal Python log rotation feature: [https://github.com/elastic/beats/blob/master/filebeat/tests/system/test\_load.py#L40](https://github.com/elastic/beats/blob/master/filebeat/tests/system/test_load.py#L40) This should automatically do the right thing. I didn't check your script in detail but you can check ours to see if there are some difference. If not needed, I would stick with rotation instead of copytruncate as this can lead to data loss (as mentioned in the logrotate docs). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 9, 2016, 6:51am UTC](https://discuss.elastic.co/t/filebeat-will-output-dumplication-when-logrotate-happened/60799/10 "2016-10-09T06:51:47Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.