Logstash randomly skipping files

[Rutger2000 (Rutger)]

I'm using logstash to ingest data into Elasticsearch and view it in Kibana. Every hour I receive multiple files in a folder which then are processed by logstash. We notice that logstash sometimes skips a file, as far as I can see, at random. The files are not added to the sincedb file either.

When I restart the logstash Docker container though, logstash does pick up the missing files and ads them to Elastic.

The link below speaks of a similar problem back in 2019 but doesn't give a clear answer. Is there anyone familiar with this problem?

File input randomly skipping files

I'm using the file input to ingest cloudflare logs into elasticsearch. I have a cronjob that runs every 10 minutes and dumps a gzip file ranging from a few hundred KB to a couple MB in size into the read path. Logstash randomly skips files for reasons yet unknown. I don't see any of the skipped filenames in debug logs. I do see an occasional plugin unrecoverable error that restarts the pipeline. My logstash config is as follows: input { file { path => "/var/spool/logstash/cloudflare*…

[Bob_Nicksic]

I have a crude hack running via cron that checks the number of files in my ingest folder. It runs at 5-59/10, because my log fetch script runs at */10 and it needs time to process any new files. If it finds more than 1, it updates a comment in the top of my config in conf.d. This triggers logstash to restart the pipeline (I'm running 3 separate pipelines), which then clears the sincedb inode reuse issue and processes the skipped file/files. I have "config.reload.automatic: true" set in my logstash.yml config file to enable the auto restart. It works for my, but YMMV

#!/bin/bash

This script checks for the existence of more than one cloudflare log in /var/spool/logstash and updates the config file to trigger a pipeline restart

It's a crude hack to work around inode reuse

Check to see if there are any unprocessed files

if ls /var/spool/logstash/*.gz 1> /dev/null 2>&1 ; then
filecount=ls /var/spool/logstash/cloudflare*.gz | wc -l

Update the top comment line in the config with the date to trigger auto reload

if [ $filecount -gt 1 ]; then
sed -i "1 s/^.*/# Config reloaded at date/" /etc/logstash/conf.d/20-cloudflare.conf
logger -t [LOGSTASH_CLOUDFLARE_PIPELINE] "Inode reuse found, logstash cloudflare pipeline restarted"
fi
fi
exit 0

[Rutger2000 (Rutger)]

Thnx for your reply Bob,

For now this does the trick. Although I can't believe that there isn't a better solution. Are the people at Logstash/Elastic aware of this problem. It's a pretty serious bug if you ask me. Anyone?

[Badger]

The inode re-use issue is well understood. There are a number of open issues that speak to it, they can be found amongst these. 211 and 251 in particular.