I am running a logstash instance looking at 10000 gz files between kbs to mb in size. The files are stored in a folder structure of
/var/log/prd/<YYYY>/<MM>/<DD>/<TYPE-3characters>/GZFILES/the_files.gz.
my file option is set to
file {
path => ["/var/log/prd/202?/??/??/???/GZFILES/*.gz"]
mode => "read"
start_position => "beginning"
codec => "json"
}
This worked until a couple days ago when it was no longer able to find the new files. When setting the day to a specific day, (remove the question marks and replace with a date and leaving the type with question marks) it is able to find the new files again. running an IO stat I see that logstash isn't reading any new files when using the wildcard format now. Is there a better way of setting the path in this use case?