# Logstash randomly skipping files **URL:** \ **Category:** Logstash\ **Created:** [January 26, 2021, 12:44pm UTC](https://discuss.elastic.co/t/logstash-randomly-skipping-files/262228 "2021-01-26T12:44:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Rutger2000](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@Rutger2000](https://discuss.elastic.co/u/Rutger2000)\ **Post date:** [January 26, 2021, 12:44pm UTC](https://discuss.elastic.co/t/logstash-randomly-skipping-files/262228/1 "2021-01-26T12:44:07Z")
I'm using logstash to ingest data into Elasticsearch and view it in Kibana. Every hour I receive multiple files in a folder which then are processed by logstash. We notice that logstash sometimes skips a file, as far as I can see, at random. The files are not added to the sincedb file either. When I restart the logstash Docker container though, logstash does pick up the missing files and ads them to Elastic. The link below speaks of a similar problem back in 2019 but doesn't give a clear answer. Is there anyone familiar with this problem? > [@File input randomly skipping files](https://discuss.elastic.co/t/file-input-randomly-skipping-files/170228): > > I'm using the file input to ingest cloudflare logs into elasticsearch. I have a cronjob that runs every 10 minutes and dumps a gzip file ranging from a few hundred KB to a couple MB in size into the read path. Logstash randomly skips files for reasons yet unknown. I don't see any of the skipped filenames in debug logs. I do see an occasional plugin unrecoverable error that restarts the pipeline. My logstash config is as follows: input { file { path =\> "/var/spool/logstash/cloudflare\*… ---
**Author:** ![Bob\_Nicksic](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bob_nicksic/32/73372_2.png) [@Bob\_Nicksic](https://discuss.elastic.co/u/Bob_Nicksic)\ **Post date:** [January 26, 2021, 1:52pm UTC](https://discuss.elastic.co/t/logstash-randomly-skipping-files/262228/2 "2021-01-26T13:52:55Z")
I have a crude hack running via cron that checks the number of files in my ingest folder. It runs at 5-59/10, because my log fetch script runs at \*/10 and it needs time to process any new files. If it finds more than 1, it updates a comment in the top of my config in conf.d. This triggers logstash to restart the pipeline (I'm running 3 separate pipelines), which then clears the sincedb inode reuse issue and processes the skipped file/files. I have "config.reload.automatic: true" set in my logstash.yml config file to enable the auto restart. It works for my, but YMMV #!/bin/bash # This script checks for the existence of more than one cloudflare log in /var/spool/logstash and updates the config file to trigger a pipeline restart # It's a crude hack to work around inode reuse # Check to see if there are any unprocessed files if ls /var/spool/logstash/\*.gz 1\> /dev/null 2\>&1 ; then filecount=`ls /var/spool/logstash/cloudflare*.gz | wc -l` # Update the top comment line in the config with the date to trigger auto reload if [$filecount -gt 1]; then sed -i "1 s/^.\*/# Config reloaded at `date`/" /etc/logstash/conf.d/20-cloudflare.conf logger -t [LOGSTASH\_CLOUDFLARE\_PIPELINE] "Inode reuse found, logstash cloudflare pipeline restarted" fi fi exit 0 ---
**Author:** ![Rutger2000](https://avatars.discourse-cdn.com/v4/letter/r/2bfe46/32.png) [@Rutger2000](https://discuss.elastic.co/u/Rutger2000)\ **Post date:** [January 27, 2021, 7:27am UTC](https://discuss.elastic.co/t/logstash-randomly-skipping-files/262228/3 "2021-01-27T07:27:26Z")
Thnx for your reply Bob, For now this does the trick. Although I can't believe that there isn't a better solution. Are the people at Logstash/Elastic aware of this problem. It's a pretty serious bug if you ask me. Anyone? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 27, 2021, 4:25pm UTC](https://discuss.elastic.co/t/logstash-randomly-skipping-files/262228/4 "2021-01-27T16:25:38Z")
The inode re-use issue is well understood. There are a number of open issues that speak to it, they can be found amongst [these](https://github.com/logstash-plugins/logstash-input-file/issues?q=is%3Aissue+is%3Aopen+inode). 211 and 251 in particular. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 24, 2021, 4:25pm UTC](https://discuss.elastic.co/t/logstash-randomly-skipping-files/262228/5 "2021-02-24T16:25:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.