In many case we do indexing from files
sometimes very large file. GBs.
What is your favorite way to index large json file to elasticsearch?
Logstash-file is very good and easy way to do that.
But logstash-file aimed to streaming small chunks.
Is there any impressively fast and safe way to index a single large json file to elasticsearch?
when I ran rally on my server rally indexed 25000 docs per second. I dont know rally's internal indexing rule, mappings.
but I only were able to hit 7000 indexing per second via logstash.
many filter workers, elasticsearch workers did not increased indexing performance. at that time elasticseaech's active bulk thread count were 0 or 1.
Test mapping is so simple. just 10 of not analyzed fields.