Hello,
I am working to import a ton of old logs into our elastic cache cluster. These are cloudtrail files so there are millions of very small files I am trying to import. When starting Logstash with an input to glob of files ~100k the cpu spins for hours w/o doing anything. Is there anyway to speed this up so the what seems like the file discovery is faster?
input {
file {
add_field => [ "import_account" , "xxx" ]
path => "/data/xxx/CloudTrail/us-east-1/2020/01/**/*.json.gz"
mode => "read"
type => "cloudtrail"
codec => "cloudtrail"
file_completed_action => "log_and_delete"
file_completed_log_path => "/data/xxx/proccessed.log"
max_open_files => 10
}
}