I have couple of millions small files, but few of them are extremely big (comparing to small ones), around 200-400 GB. Pipeline is set to fetch data from each file (1 file -> 1 event in Elasticsearch). It's working ok, but when it comes to large file it takes around 2-3, sometimes even more hours to…

Slow performance on big unstructured file

Badger September 22, 2021, 5:31pm 11

Use a multiline codec. An example is here.

Elasticsearch is a trademark of Elasticsearch BV, registered in the U.S. and in other countries
Trademarks
Terms
Privacy
Brand
Code of Conduct

Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.