I have couple of millions small files, but few of them are extremely big (comparing to small ones), around 200-400 GB. Pipeline is set to fetch data from each file (1 file -> 1 event in Elasticsearch). It's working ok, but when it comes to large file it takes around 2-3, sometimes even more hours to…

Slow performance on big unstructured file

Badger September 22, 2021, 5:31pm 11

Use a multiline codec. An example is here.

Topic		Replies	Views
Slow processing from s3 for larger gzip file Logstash	1	622	February 1, 2018
Small Scale elasticsearch implementation Elasticsearch	2	445	July 6, 2017
Multiline codec and csv filter performance Logstash	4	396	August 19, 2022
Logstash to process multiple files of large size? Logstash	1	269	April 19, 2022
Logstash - Slow data processing Logstash	19	112	December 3, 2024

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Slow performance on big unstructured file

Related topics