# Entire log is read when it changes **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [June 25, 2018, 3:06am UTC](https://discuss.elastic.co/t/entire-log-is-read-when-it-changes/137175 "2018-06-25T03:06:48Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![joshsmoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshsmoore/32/31581_2.png) [@joshsmoore](https://discuss.elastic.co/u/joshsmoore)\ **Post date:** [June 25, 2018, 3:06am UTC](https://discuss.elastic.co/t/entire-log-is-read-when-it-changes/137175/1 "2018-06-25T03:06:48Z")
Hi I am using filebeats and I am having some problems with my backup log file. This file is written to once a day. The problem is that when this log file is updated the entire file is read and sent to logstash twice. The other log files appear to be sending data correctly, however, they are written to much more often. Any ideas on why this is happening or how to debug it? Incase it helps here is the prospector snippet: - paths: - /var/log/backup.log fields: type: backup server: rg\_u16\_prod\_db\_slave env: rg\_production application\_env: production chef\_roles: ["server", "mysql\_db\_slave"] scan\_frequency: "60s" backoff: "1s" ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [June 25, 2018, 5:22pm UTC](https://discuss.elastic.co/t/entire-log-is-read-when-it-changes/137175/2 "2018-06-25T17:22:12Z")
Hi @joshsmoore, How is your backup file written? New content is appended or content is replaced every time it is written? What version of filebeat are you using? You can check in the filebeat logs if you see any issue regarding this file. ---
**Author:** ![joshsmoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshsmoore/32/31581_2.png) [@joshsmoore](https://discuss.elastic.co/u/joshsmoore)\ **Post date:** [June 26, 2018, 6:30pm UTC](https://discuss.elastic.co/t/entire-log-is-read-when-it-changes/137175/3 "2018-06-26T18:30:03Z")
I think it is just appended because the inode number stays the same. I do not see any problems in the log and I am running filebeats 6.3.0 ---
**Author:** ![joshsmoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshsmoore/32/31581_2.png) [@joshsmoore](https://discuss.elastic.co/u/joshsmoore)\ **Post date:** [June 29, 2018, 3:11am UTC](https://discuss.elastic.co/t/entire-log-is-read-when-it-changes/137175/4 "2018-06-29T03:11:56Z")
I tried to get some more information by deleting the old log and just looking at the new entries. However, I am still reading in 6000 lines everytime the file is changed. The interesting thing is that log items are being added that do not exist in the file. So I am wondering if somehow these lines are getting stuck in logstash. Where the lines is written to elasticsearch but logstash does not think it has written it. Is this possible? ---
**Author:** ![joshsmoore](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joshsmoore/32/31581_2.png) [@joshsmoore](https://discuss.elastic.co/u/joshsmoore)\ **Post date:** [July 3, 2018, 2:04am UTC](https://discuss.elastic.co/t/entire-log-is-read-when-it-changes/137175/5 "2018-07-03T02:04:42Z")
I found the problem. The utility that was writing the log copied the log to a new file truncated and copied back. That was the problem. Thanks, ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 31, 2018, 2:04am UTC](https://discuss.elastic.co/t/entire-log-is-read-when-it-changes/137175/6 "2018-07-31T02:04:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.