# Why is filebeat reading log files over and over again **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 23, 2020, 11:06am UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654 "2020-03-23T11:06:25Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![SjonnieW](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@SjonnieW](https://discuss.elastic.co/u/SjonnieW)\ **Post date:** [March 23, 2020, 11:06am UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654/1 "2020-03-23T11:06:25Z")
Hello all I know this was posted before, only i never read a satifying answer\solution. I was advised by my user succes manager to post the problem here Using a windows10 environment (also tried on Linux) I am using a simple configuration to read a log file with logbeat. To start logstash i use the command .\bin\logstash -f .\config\sample.conf Sample.conf: input { beats { port =\> 5044 } } filter { grok { match =\> [ "message", "%{TIMESTAMP\_ISO8601:timestamp\_string} %{SPACE}%{GREEDYDATA:line}" ] } mutate { remove\_field =\> [message, timestamp\_string] } } output { elasticsearch { hosts =\> ["[http://localhost:9200](http://localhost:9200)"] } stdout { codec =\> rubydebug } } I start filebeat with the command .\filebeat Filebeat.yml: filebeat.inputs: - type: log enabled: true paths: - ./sample.log output.logstash: hosts: ["localhost:5044"] Sample.log contains 14 records What happenes is that the log file is being read and send over and over again wich will give a lot of duplicates. I found a way to avoid duplicates with the use of a fingerprint but that is not what i want. I want the logfile only being updated by filebeat when a change happenes in the file and not being read all over again. Also tried ignore\_older: 5s, but it gave the same results. In the registry file data.json offset is constantly set to 0 question: Why are basic functions of filebeat not working (what am i missing) ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 23, 2020, 11:30am UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654/2 "2020-03-23T11:30:13Z")
Is it reading the log from a local file or a network drive? Is the file bring appended to or copied into place? ---
**Author:** ![SjonnieW](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@SjonnieW](https://discuss.elastic.co/u/SjonnieW)\ **Post date:** [March 23, 2020, 1:15pm UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654/3 "2020-03-23T13:15:12Z")
The whole setup is on one machine including the log file. I tried it both ways, copying the file and appending to a file with echo -n "text" \>\> /{path}/sample.log ---
**Author:** ![SjonnieW](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@SjonnieW](https://discuss.elastic.co/u/SjonnieW)\ **Post date:** [March 26, 2020, 4:19pm UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654/4 "2020-03-26T16:19:39Z")
Solved. Its not filebeat itself causing the problem but logstash Cause: Buggy logstash 7.5.2 Solution: Replace logstash 7.5.2 with 7.5.1 or 7.6.1 Or fully upgrade to 7.6.1 ---
**Author:** ![brendanlynch](https://avatars.discourse-cdn.com/v4/letter/b/eada6e/32.png) [@brendanlynch](https://discuss.elastic.co/u/brendanlynch)\ **Post date:** [April 21, 2020, 1:40pm UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654/5 "2020-04-21T13:40:56Z")
Hi, Can you expand on how logstash bug contributed to the duplication? I very new to both filebeat and logstash and running the same very simple config like in this thread. I send file through and it completes all the events. I have file updated to include a couple new records and it results in all the records from the top of the file getting written again ---
**Author:** ![aviationfan](https://avatars.discourse-cdn.com/v4/letter/a/c37758/32.png) [@aviationfan](https://discuss.elastic.co/u/aviationfan)\ **Post date:** [April 21, 2020, 2:44pm UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654/6 "2020-04-21T14:44:29Z")
Just had this exact same problem today. My server needed an restart after some updates so I manually stopped all the Elastic related services, restarted the machine, then brought all the services back up. For some reason it grabbed every log file and started indexing them again even though they all had been read in the past. What am I missing here? I thought the design of the registry was to take this into account and know that the files were already read and indexed. `2020-04-21T07:45:27.035-0600 INFO registrar/registrar.go:145 Loading registrar data from /usr/local/var/lib/filebeat/registry/filebeat/data.json` These files are copied from my raspberry pi once a day and then indexed so they are not changing over time. To fix this I now have to delete the index for 2020, restore all the daily log files, and re-index everything from scratch. ---
**Author:** ![SjonnieW](https://avatars.discourse-cdn.com/v4/letter/s/ea666f/32.png) [@SjonnieW](https://discuss.elastic.co/u/SjonnieW)\ **Post date:** [April 22, 2020, 6:37am UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654/7 "2020-04-22T06:37:52Z")
I don't no exactly how logstash is related to the problem i was using version 7.5.2 then I tried some combi's with versions leaving me 1 conclusion elastic 7.6.1 kibana 7.6.1 ; logstash 7.6.1 filebeat 7.5.2 no duplication problm elastic 7.6.1 kibana 7.6.1 ; logstash 7.5.2 filebeat 7.6.1 duplication problem elastic 7.6.1 kibana 7.6.1 ; logstash en filebeat 7.5.2 duplication problem elastic 7.5.2 kibana 7.5.2 ; logstash en filebeat 7.6.1 no duplication problem elastic 7.6.1 kibana 7.6.1 ; logstash en filebeat 7.5.1 no duplication problem elastic 7.5.1 kibana 7.5.1 ; logstash en filebeat 7.6.1 no duplication problem so try replacing logstash with another version and find out if the problem still exists ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 20, 2020, 6:37am UTC](https://discuss.elastic.co/t/why-is-filebeat-reading-log-files-over-and-over-again/224654/8 "2020-05-20T06:37:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.