# Logstash read the entire file again whenever pipeline reload **URL:** \ **Category:** Logstash\ **Created:** [March 12, 2020, 5:00am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247 "2020-03-12T05:00:57Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 12, 2020, 5:00am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/1 "2020-03-12T05:00:57Z")
Hi all, What I want is read all the file in a directory and tail for new change and discover new files Btw, the sincedb file I specified is always 0byte, nothing ever write My config: > input { > file { > path =\> "/home/datareceive/elkGetLock/export\*" > start\_position =\> "beginning" > sincedb\_path =\> "/home/esdteam/moodleData/getlock.sincedb" > close\_older =\> "15 s" > } > } Thank you. ---
**Author:** ![Nicolas\_Aizier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nicolas_aizier/32/13631_2.png) [@Nicolas\_Aizier](https://discuss.elastic.co/u/Nicolas_Aizier)\ **Post date:** [March 12, 2020, 5:34am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/2 "2020-03-12T05:34:51Z")
Couldn't it be some access permission to this file ? Have you try to read/write in it with the logstash user ? ---
**Author:** ![pariksh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pariksh/32/46188_2.png) [@pariksh](https://discuss.elastic.co/u/pariksh)\ **Post date:** [March 12, 2020, 5:59am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/3 "2020-03-12T05:59:01Z")
> [@mmk1995](#): > > . Remove start\_position =\> "beginning" ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 12, 2020, 6:11am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/4 "2020-03-12T06:11:54Z")
i have setfacl to the running user and should not be the permission issue ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 12, 2020, 6:12am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/5 "2020-03-12T06:12:25Z")
withont start\_position =\> "beginning", I am not able to read the old files within the directory ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 13, 2020, 2:33am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/6 "2020-03-13T02:33:16Z")
Anyone could help please? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2020, 2:38am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/7 "2020-03-13T02:38:47Z")
How are the files updated? Is there anything in the logs? ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 13, 2020, 3:59am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/8 "2020-03-13T03:59:56Z")
there is an scp job copying files to the directory whenever there is updates. which log you mean? logstash? what log message you are looking for? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2020, 6:23am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/9 "2020-03-13T06:23:35Z")
Logstash tracks files by inode. If you are copying over a file that has increased in size it will appear as a new file even if it has the same name as it gets a new inode assigned. This is why the file is reread repeatedly. ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 13, 2020, 7:04am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/10 "2020-03-13T07:04:48Z")
I take your point, I have handled this using document\_id. So sincedb only works when tailing a particular file? the sincedb is always 0byte as i observe. Or how it actually works? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2020, 8:02am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/11 "2020-03-13T08:02:18Z")
The file input expects data to be appended to local files that keep the inode. Since dub should be populated but I wonder if your close\_older config affects this. Remove this and see what happens. ---
**Author:** ![mmk1995](https://avatars.discourse-cdn.com/v4/letter/m/8797f3/32.png) [@mmk1995](https://discuss.elastic.co/u/mmk1995)\ **Post date:** [March 13, 2020, 9:50am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/12 "2020-03-13T09:50:45Z")
My ES cluster is currently down due to disk failure, not sure if it is related the ELK itself and still investigating, I cannot test what you said for now. If possible, please advice if there is any cases from other users if ELK application causing any disk failure in the past, mine ELK stack is version 6.5.4 Back to the question, it doesn't make sense if it is related to close\_older, because if I do not close the file, it would eventually reach the file open limit as the number of file is increasing. My initiation is to open the file again for any new data or updated file modified time and start from the new line instead of the beginning of the file which is depending on the sincedb. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 13, 2020, 10:34am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/13 "2020-03-13T10:34:31Z")
As far as I know Logstash does not work that way so getting Logstash to behave like you describe might not be possible. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 10, 2020, 10:34am UTC](https://discuss.elastic.co/t/logstash-read-the-entire-file-again-whenever-pipeline-reload/223247/14 "2020-04-10T10:34:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.