# File input plugin does not read files **URL:** \ **Category:** Logstash\ **Created:** [April 1, 2019, 11:39am UTC](https://discuss.elastic.co/t/file-input-plugin-does-not-read-files/174769 "2019-04-01T11:39:13Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Nathan\_Sowatskey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathan_sowatskey/32/42778_2.png) [@Nathan\_Sowatskey](https://discuss.elastic.co/u/Nathan_Sowatskey)\ **Post date:** [April 1, 2019, 11:39am UTC](https://discuss.elastic.co/t/file-input-plugin-does-not-read-files/174769/1 "2019-04-01T11:39:13Z")
I have a file input plugin configured as below. The first time I ran Logstash with this configuration, _I think_ it read the files in question and created an index. Subsequently, it definitely has not. A similar configuration without the `file_completed_action` definitely did work, but it also deletes the files after reading them, by default. That is not what I wanted, so hence the need to add an explicit `file_completed_action` as `log`, and so also the `file_completed_log_path`. I _suspect_ that the problem lies in the logic of not reading files that have already been read. Note that I have `file_completed_log_path` configured below. That file is empty though. The behaviour here is as though the plugin "knows" that it has read the files in question. I can't, though, see which configuration/log file, where, has the list of files that this plugin thinks it has read already. It does not appear to be the file configured as `file_completed_log_path`, as that file is empty. I have tried `touch`ing the files in question, to see if a date change would trigger a read, but that did not seem to make any difference either. I know that the files are not being read as there is no index corresponding to those files. Any ideas please? Many thanks Nathan ````auto input { file { path => ["/.../docker_inspect_k8s*.log"] type => "rca_docker_inspect" codec => "line" mode => "read" file_completed_action => "log" file_completed_log_path => "logged_file.list" } ...``` ```` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 1, 2019, 12:35pm UTC](https://discuss.elastic.co/t/file-input-plugin-does-not-read-files/174769/2 "2019-04-01T12:35:37Z")
> [@Nathan\_Sowatskey](#): > > The behaviour here is as though the plugin "knows" that it has read the files in question. Yes, it does. It saves state in the "sincedb". This is maintained in-memory and peristed to sincedb\_path. If you want the state reset every time you restart logstash then set ``` sincedb_path => "/dev/null" ``` ---
**Author:** ![Nathan\_Sowatskey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathan_sowatskey/32/42778_2.png) [@Nathan\_Sowatskey](https://discuss.elastic.co/u/Nathan_Sowatskey)\ **Post date:** [April 5, 2019, 2:56pm UTC](https://discuss.elastic.co/t/file-input-plugin-does-not-read-files/174769/3 "2019-04-05T14:56:07Z")
> [@Badger](#): > > sincedb\_path First of all, thank you, and apologies for the delay in following up. I _think_ that this suggestion did make a difference, but I am still seeing unexpected behaviour. I now have: ```auto input { file { id => "rca_docker_inspect" path => ["/.../docker_inspect_k8s*.log"] type => "rca_docker_inspect" codec => "line" mode => "read" file_completed_action => "log" file_completed_log_path => "logged_file.list" sincedb_path => "/dev/null" } ... output { elasticsearch { hosts => ["localhost:9200"] index => "rca_docker_inspect-%{+YYYY.MM.dd}" } ... ``` BUT, the index only appears in ES _after_ I stop Logstash with a Ctrl-C. When Logstash is running, I don't see the index at: [http://localhost:9200/\_aliases?pretty=true](http://localhost:9200/_aliases?pretty=true) So, I now have something that is repeatable, but weird. Regards Nathan ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 5, 2019, 3:17pm UTC](https://discuss.elastic.co/t/file-input-plugin-does-not-read-files/174769/4 "2019-04-05T15:17:30Z")
That would suggest that the codec is never seeing a newline to tell it to flush an event, so it only flushes on shutdown. ---
**Author:** ![Nathan\_Sowatskey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathan_sowatskey/32/42778_2.png) [@Nathan\_Sowatskey](https://discuss.elastic.co/u/Nathan_Sowatskey)\ **Post date:** [April 5, 2019, 3:50pm UTC](https://discuss.elastic.co/t/file-input-plugin-does-not-read-files/174769/5 "2019-04-05T15:50:23Z")
Yes, the mode is 'read' though, so it should just read the files and get on with it. What I am seeing does not make sense. I am still missing something I suspect. I am also a bit baffled that it is so tricky to just read some files. This seems like a core use case for Logstash/ES, but it is proving to be much harder than I expected. Regards Nathan ---
**Author:** ![Nathan\_Sowatskey](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nathan_sowatskey/32/42778_2.png) [@Nathan\_Sowatskey](https://discuss.elastic.co/u/Nathan_Sowatskey)\ **Post date:** [April 5, 2019, 6:12pm UTC](https://discuss.elastic.co/t/file-input-plugin-does-not-read-files/174769/6 "2019-04-05T18:12:11Z")
OK, though I can see that many files are being processed, within which there are many loge entries, so there must be a new line in somewhere ... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 3, 2019, 6:22pm UTC](https://discuss.elastic.co/t/file-input-plugin-does-not-read-files/174769/7 "2019-05-03T18:22:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.