Logstash randomly skipping files

I'm using logstash to ingest data into Elasticsearch and view it in Kibana. Every hour I receive multiple files in a folder which then are processed by logstash. We notice that logstash sometimes skips a file, as far as I can see, at random. The files are not added to the sincedb file either.

When I restart the logstash Docker container though, logstash does pick up the missing files and ads them to Elastic.

The link below speaks of a similar problem back in 2019 but doesn't give a clear answer. Is there anyone familiar with this problem?

I have a crude hack running via cron that checks the number of files in my ingest folder. It runs at 5-59/10, because my log fetch script runs at */10 and it needs time to process any new files. If it finds more than 1, it updates a comment in the top of my config in conf.d. This triggers logstash to restart the pipeline (I'm running 3 separate pipelines), which then clears the sincedb inode reuse issue and processes the skipped file/files. I have "config.reload.automatic: true" set in my logstash.yml config file to enable the auto restart. It works for my, but YMMV


This script checks for the existence of more than one cloudflare log in /var/spool/logstash and updates the config file to trigger a pipeline restart

It's a crude hack to work around inode reuse

Check to see if there are any unprocessed files

if ls /var/spool/logstash/*.gz 1> /dev/null 2>&1 ; then
filecount=ls /var/spool/logstash/cloudflare*.gz | wc -l

Update the top comment line in the config with the date to trigger auto reload

if [ $filecount -gt 1 ]; then
sed -i "1 s/^.*/# Config reloaded at date/" /etc/logstash/conf.d/20-cloudflare.conf
logger -t [LOGSTASH_CLOUDFLARE_PIPELINE] "Inode reuse found, logstash cloudflare pipeline restarted"
exit 0

Thnx for your reply Bob,

For now this does the trick. Although I can't believe that there isn't a better solution. Are the people at Logstash/Elastic aware of this problem. It's a pretty serious bug if you ask me. Anyone?

The inode re-use issue is well understood. There are a number of open issues that speak to it, they can be found amongst these. 211 and 251 in particular.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.