Hi
Context: I am processing a dataset that contains a lot of subfolder's (2359 in total), when I run my logstash config on for example 5 subfolder's everything is getting processed correctly but on the full dataset this doesn't work
Input config:
file {
# xml logs
path => "/var/log/logstash/**/*.paldata"
start_position => "beginning"
sincedb_path => "/dev/null"
codec => multiline {
pattern => "</data>"
negate => true
what => "next"
auto_flush_interval => 1
}
mode => "read"
type => "xml"
file_completed_action => "log"
file_completed_log_path => "/var/log/logstash/file_completed_audit"
}
when I check on the file_completed_log_path, I see no files that have been processed so it looks like it doesn't even read it
I checked in the folder and the subfolders are present
I increased my memory to give it more room but this didn't work either
Any ideas what could be the cause of this?