Logstash CSV output opening/closing file bottleneck

Elastic Stack Logstash
1

I am running logstash to parse many files with output to a single CSV file. This is working ok but is taking long with the bottleneck being the opening file/closing file. Im i missing something? Any faster way of doin this?

Input as below;

input {

    file {

    path =>"C:/ELK/LOGS/LX/dec/*.txt"

    start_position =>"beginning"

    sincedb_path=>"NUL"

                     

    }

}

Output,

output {

   

    csv {

        fields => ["timestamp","field1","field 2","field 3","field4"]

        path => "C:/ELK/LOGS/LX/file.csv"

        

    }

}

When i run the logstash, you can see the time stamps of the opening file/closing file

[2022-01-15T04:26:06,342][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Closing file C:/ELK/LOGS/LX/file.csv
[2022-01-15T04:26:14,563][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Opening file {:path=>"C:/ELK/LOGS/LX/file.csv"}
[2022-01-15T04:29:08,517][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Closing file C:/ELK/LOGS/LX/file.csv
[2022-01-15T04:29:11,458][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Opening file {:path=>"C:/ELK/LOGS/LX/file.csv"}
[2022-01-15T04:30:22,281][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Closing file C:/ELK/LOGS/LX/file.csv
[2022-01-15T04:30:29,816][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Opening file {:path=>"C:/ELK/LOGS/LX/file.csv"}
[2022-01-15T04:31:03,544][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Closing file C:/ELK/LOGS/LX/file.csv
[2022-01-15T04:31:08,105][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Opening file {:path=>"C:/ELK/LOGS/LX/file.csv"}
[2022-01-15T04:33:11,831][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Closing file C:/ELK/LOGS/LX/file.csv
[2022-01-15T04:34:57,481][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Opening file {:path=>"C:/ELK/LOGS/LX/file.csv"}
[2022-01-15T04:36:01,097][INFO ][logstash.outputs.csv     ][main][62c6c42308330edea714c856c9da89029681556c2a28c1ae2345c8063277abcc] Closing file C:/ELK/LOGS/LX/file.csv}

Any way to make this more efficient?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.