Have ~ 50K files in single directory( ~200Gb ) of logs. trying to process to parse and add them to elasticsearch my config: input { file { path => "/mnt/storage/*.txt" sincedb_path => "/dev/null" codec => "json_lines" start_position => "beginning" ignore_older => 123123123…

One time batch processing big number of files

guyboertje (Guy Boertje) April 6, 2016, 1:10pm 11

Glad to help, happy stashing

Topic		Replies	Views
Slow performance on big unstructured file Logstash	10	759	October 20, 2021
How to process multiple files in logstash Logstash	5	2199	September 14, 2017
Elasticsearch Logstash importing very large JSON files Logstash	4	4835	July 6, 2017
Loading many (big) json files into elasticsearch Logstash	14	12945	May 23, 2018
Parse json file with 100k/1 million records? Logstash	8	1327	December 2, 2019

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

One time batch processing big number of files

Related topics