# Fastest way to import billions of documents? **URL:** \ **Category:** Elasticsearch\ **Created:** [April 5, 2017, 1:49pm UTC](https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317 "2017-04-05T13:49:44Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![jeroen1](https://avatars.discourse-cdn.com/v4/letter/j/47e85d/32.png) [@jeroen1](https://discuss.elastic.co/u/jeroen1)\ **Post date:** [April 5, 2017, 1:49pm UTC](https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317/1 "2017-04-05T13:49:44Z")
I want to import huge datasets in a single node ESv5.3 setup (billions of documents). Right now I'm using the latest version of ElasticDump, that uses the bulk API. Imporing takes 24h+. I see no obvious reason for this: - I'm using a 24 core server. According to 'top' the load is 400-500% (4/5 cores fully used). I expect something close to 2400%. - I'm using a local SSD RAID set. According to 'iotop' the speed is 10-40 MByte/s. I expect 100s of MBytes/s or more (disk array bechmark \> 1 GByte/s) So my server is doing near to nothing and I'm just waiting... Things I've already tried without (significant) success: - JVM heap size = 31G (server with massive amount or RAM) - Increasing the number of ElasticDump sockets or documents per go - Increasing index interval during import. - Increasing the thread pool queue size. The only thing that speeds up the process is pretty stupid: run several ElasticDumps in parallel... Anyone with tips 'n' tricks about how to improve the import speed by using other settings or tools? Thanks! ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 5, 2017, 2:11pm UTC](https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317/2 "2017-04-05T14:11:05Z")
I expect ElasticDump is running single `_bulk` inserts which are internally parallelized into once per shard. Given the default number of shards in 5 I fully expect a load average of around 5. Step one is to use more parallel bulk loads. If that means passing something to ElasticDump or running multiple instances, fine. I figure that is an ElasticDump concern and you are on an Elasticsearch board, so you want to know about Elasticsearch things. The usual strategy is indeed multiple `_bulk`s in parallel. You might also play with the size of the `_bulk`s. I don't know what ElasticDump is doing - but it is usually a good practice to shoot for 10mb or 20mb or something on each `_bulk` request. You can optimize up or down from there. It is also worth looking at the mappings that your documents are making. You should set up the mapping before you import so there is no contention around mapping updates. You can reduce overhead of each document significantly (or increase it) by changing how string fields are analyzed. If happen to have any shape fields there are good settings to play with for that too. Finally: you might get some performance benefit from running more shards. Probably some. After that you can use the `_shrink` API to reduce the number of shards so you don't suffer any performance penalty while querying. ---
**Author:** ![jeroen1](https://avatars.discourse-cdn.com/v4/letter/j/47e85d/32.png) [@jeroen1](https://discuss.elastic.co/u/jeroen1)\ **Post date:** [April 5, 2017, 2:50pm UTC](https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317/3 "2017-04-05T14:50:39Z")
Thanks for your quick reply! I forgot to mention: I've tested using 12 and 24 shards (1 per CPU core, in and excluding HyperThreading) so expecting 1200-2400% load. Mappings, analyzers and filters are already highly optimized / stripped down. Two questions based on your reply: - You're mentioning 10 or 20 MByte per \_bulk request. Is there some way I can monitor what will work best for my specific setup? Perhaps some scripts / tools that can perform a benchmark for optimal values? - You're mentioning that multiple \_bulks in parallel might be a or the way to go. Any idea why the full potential of an import is limited when using a single \_bulk process? I guess that importing huge datasets in ElasticSearch is not uncommon to I expect it to be (quite) optimized. Thanks for your thoughts! ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 5, 2017, 3:59pm UTC](https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317/4 "2017-04-05T15:59:25Z")
> [@jeroen1](#): > > You're mentioning 10 or 20 MByte per \_bulk request. Is there some way I can monitor what will work best for my specific setup? Perhaps some scripts / tools that can perform a benchmark for optimal values? That is something Elasticsearch expects the importing application to perform. So I'd look at ElasticDump and see if it has logging for that. > [@jeroen1](#): > > Any idea why the full potential of an import is limited when using a single \_bulk process? I'm not sure. Ask the ElasticDump folks. Often when folks import a ton of data into Elasticsearch they are doing it from many nodes so they get the natural parallelization from that. ---
**Author:** ![jeroen1](https://avatars.discourse-cdn.com/v4/letter/j/47e85d/32.png) [@jeroen1](https://discuss.elastic.co/u/jeroen1)\ **Post date:** [April 5, 2017, 5:18pm UTC](https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317/5 "2017-04-05T17:18:18Z")
I'm not bound to ElasticDump btw. All suggestions that speed up the import - both ES settings and other ways of importing - are welcome. The only reason for that I'm using ElasticDump now is that I read posts telling me that's an easy to use and fast tool ofr bulk imports. If there's a ES native way or supermagic curl options I'm also perfectly fine with that 😉 ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [April 5, 2017, 5:42pm UTC](https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317/6 "2017-04-05T17:42:49Z")
There isn't a "native" way, really. There are filebeat and logstash, both of which are things actively worked on by Elastic folks that stick files into Elasticsearch but they are both more organized around many servers loading into one Elasticsearch so they only go with a single bulk. Elasticsearch doesn't support any kind of streaming loads like would be required to make a supermagic curl command work. We've talked about it but never done it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 3, 2017, 5:42pm UTC](https://discuss.elastic.co/t/fastest-way-to-import-billions-of-documents/81317/7 "2017-05-03T17:42:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.