Logstash file input very slow with lots of files


I am working to import a ton of old logs into our elastic cache cluster. These are cloudtrail files so there are millions of very small files I am trying to import. When starting Logstash with an input to glob of files ~100k the cpu spins for hours w/o doing anything. Is there anyway to speed this up so the what seems like the file discovery is faster?

    input {
       file {
            add_field => [ "import_account" , "xxx" ]
            path => "/data/xxx/CloudTrail/us-east-1/2020/01/**/*.json.gz"
            mode => "read"
            type => "cloudtrail"
            codec => "cloudtrail"
            file_completed_action => "log_and_delete"
            file_completed_log_path => "/data/xxx/proccessed.log"
            max_open_files => 10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.