I have couple of millions small files, but few of them are extremely big (comparing to small ones), around 200-400 GB. Pipeline is set to fetch data from each file (1 file -> 1 event in Elasticsearch). It's working ok, but when it comes to large file it takes around 2-3, sometimes even more hours to…

Slow performance on big unstructured file

Badger September 22, 2021, 5:31pm 11

Use a multiline codec. An example is here.

Topic		Replies	Views
How can I improve Logstash performance? Logstash	25	7148	August 24, 2018
Small Scale elasticsearch implementation Elasticsearch	2	445	July 6, 2017
Multiline codec and csv filter performance Logstash	4	395	August 19, 2022
Logstash to process multiple files of large size? Logstash	1	268	April 19, 2022
Speed up the pipeline-2 Logstash	5	2340	February 15, 2018

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.

Slow performance on big unstructured file

Related Topics