# Loading 500 GB of data / 3500000000 documents to ES cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [June 29, 2018, 10:32am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935 "2018-06-29T10:32:22Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [June 29, 2018, 10:32am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/1 "2018-06-29T10:32:22Z")
Greetings! I have a small cluster with following setup: **Hardware:** 2x 64 GB RAM, 16 cores CPU, 4x2TB HDD configured in RAID10. There is 1 Gbps connection between the nodes. **Configuration** : **elasticsearch.yml node1.test:** cluster.name: mycluster node.name: node1.test node.master: true node.data: true #discovery.zen.ping.multicast.enabled: false discovery.zen.ping.unicast.hosts: [node2.test] discovery.zen.minimum\_master\_nodes: 2 path.data: /elasticsearch-data/ path.logs: /var/log/elasticsearch index.store.type: niofs network.host: 192.168.1.2 http.port: 9200 xpack.security.enabled: false **elasticsearch.yml node2.test:** cluster.name: mycluster node.name: node1.test node.master: true node.data: true discovery.zen.ping.unicast.hosts: [node1.test] discovery.zen.minimum\_master\_nodes: 2 path.data: /elasticsearch-data/ path.logs: /var/log/elasticsearch index.store.type: niofs network.host: 192.168.1.3 http.port: 9200 xpack.security.enabled: false I am limiting my JVM heap to have only 32 GB per node (as ES recommends to leave 50% of the physical RAM). I have 500 GB of json files with total number of new lines 3,500,000,000 (new line delimited json files, 30,000,000 lines, 2.5GB-4GB in size each). **First time I have used the tool to load my data:** esbulk\_0.5.0\_amd64.deb with parameters: esbulk -size 1000 -w 16 -verbose -server 192.168.1.2:9200 -u -index bigindex myjsonfiles.json it was loaded into 5 default shards without replicas. Tool applies: applied setting: {"index": {"refresh\_interval": "-1"}} with status 200 OK The load of that data was over night lasted approx 12h, but out 3,500,000,000 I have managed to load only 2,000,000,000. **I have noticed during the load following error occurring all the time:** 2018/06/29 12:15:25 error during bulk operation, check error details, try less workers (lower -w value) or increase thread\_pool.bulk.queue\_size in your nodes I have tried to limit the processing in the tool to two workers with size 50 and the same error was happening (not to mention that the data load will take forever). I have tried to split the files into smaller ones (100000 lines, approx 8MB each), and still the same problem was experienced. What would you recommend to load that 500 GB worth data in a reasonable speed and time? Should I tweak the default number of shards and (I have read about one recommendation that I should not go above 20-25 shards per 1GB of JVM memory HEAP)? How I can approach the problem of loading such data in a different way? Thank you for your time in advance! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2018, 10:44am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/2 "2018-06-29T10:44:39Z")
Have you followed [these guidelines](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/tune-for-indexing-speed.html) and set `index.merge.scheduler.max_thread_count` [to an appropriate value](https://www.elastic.co/guide/en/elasticsearch/reference/6.3/index-modules-merge.html)? why are you setting `index.store.type: niofs`? ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [June 29, 2018, 11:28am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/3 "2018-06-29T11:28:00Z")
I have configured: "max\_thread\_count": "1" "settings": { "index": { "refresh\_interval": "-1", "number\_of\_shards": "5", "provided\_name": "bigindex", "merge": { "scheduler": { "max\_thread\_count": "1" } }, "creation\_date": "1530270568301", "number\_of\_replicas": "1", "uuid": "QLplGm7bQmyc3Y1N-JVR-A", "version": { "created": "6020499" } } } } niofs was set because I thought that it is good idea and will increase performance after reading: [https://thoughts.t37.net/how-we-reindexed-36-billions-documents-in-5-days-within-the-same-elasticsearch-cluster-cd9c054d1db8](https://thoughts.t37.net/how-we-reindexed-36-billions-documents-in-5-days-within-the-same-elasticsearch-cluster-cd9c054d1db8) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2018, 11:54am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/4 "2018-06-29T11:54:02Z")
Do you have monitoring installed? What does CPU usage, disk I/O and iowait look like while you are indexing? ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [June 29, 2018, 12:02pm UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/5 "2018-06-29T12:02:12Z")
At this stage I was just monitoring with htop, kibana monitoring and iostat Kibana shows indexing speeds approx. 50k/s - 60k/s Result from iostat during loading: > avg-cpu: %user %nice %system %iowait %steal %idle > 8.77 0.00 0.56 8.96 0.00 81.70 > > Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r\_await w\_await svctm %util > sda 0.00 94.00 0.00 121.00 0.00 3260.00 53.88 0.99 7.80 0.00 7.80 5.39 65.20 > sdb 0.00 94.00 0.00 121.00 0.00 3260.00 53.88 0.90 7.11 0.00 7.11 5.06 61.20 > md0 0.00 0.00 0.00 369.00 0.00 6528.00 35.38 0.00 0.00 0.00 0.00 0.00 0.00 > sdc 0.00 110.00 0.00 106.00 0.00 3192.00 60.23 0.61 5.40 0.00 5.40 4.42 46.80 > sdd 0.00 110.00 0.00 106.00 0.00 3192.00 60.23 0.56 4.94 0.00 4.94 3.85 40.80 > dm-0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-1 0.00 0.00 0.00 4.00 0.00 16.00 8.00 0.11 28.00 0.00 28.00 24.00 9.60 > dm-2 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 > dm-4 0.00 0.00 0.00 349.00 0.00 6512.00 37.32 3.50 9.78 0.00 9.78 2.85 99.60 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2018, 12:05pm UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/6 "2018-06-29T12:05:00Z")
If you look at the last line there it looks like you may be limited by the speed of your storage as it looks like it shows a 99.6% utilisation. To improve performance you may therefore need to invest in faster storage to scale out the cluster. You could also try using RAID0 instead of RAID10. ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [June 29, 2018, 12:06pm UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/7 "2018-06-29T12:06:51Z")
Right, good catch. I will strip the raid10 and go with raid0 on all drives. ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [July 24, 2018, 11:32am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/8 "2018-07-24T11:32:08Z")
I am using RAID0 on 4 spinning drives. Performance is noticeable with stability. Side note: Before I was using esbulk, right now I am using esload for placing data into ES cluster. Both of them are in the github. ---
**Author:** ![zqc0512](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zqc0512/32/32141_2.png) [@zqc0512](https://discuss.elastic.co/u/zqc0512)\ **Post date:** [July 25, 2018, 2:30am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/9 "2018-07-25T02:30:52Z")
> [@pastechecker](#): > > iowait iowait is too high. try with **asnyc** ---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [July 25, 2018, 7:28am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/10 "2018-07-25T07:28:04Z")
With the tool esload I was able to load in 12h approximately 500GB/2.2b of documents (New Line Json Delimited). INFO loader.py:index:105 uploaded 80000 documents (errors: n/a) (6.14MB) in 1.10s (5.60MB/s) INFO loader.py:index:105 uploaded 80000 documents (errors: n/a) (6.09MB) in 0.94s (6.47MB/s) INFO loader.py:index:105 uploaded 80000 documents (errors: n/a) (6.20MB) in 0.96s (6.47MB/s) INFO loader.py:index:105 uploaded 80000 documents (errors: n/a) (6.13MB) in 0.99s (6.20MB/s) INFO loader.py:index:105 uploaded 80000 documents (errors: n/a) (6.17MB) in 0.88s (7.02MB/s) INFO loader.py:index:105 uploaded 80000 documents (errors: n/a) (6.16MB) in 2.23s (2.76MB/s) INFO loader.py:index:105 uploaded 80000 documents (errors: n/a) (6.12MB) in 0.92s (6.67MB/s) Start after work, over night it will do the job. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 22, 2018, 7:28am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/11 "2018-08-22T07:28:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 22, 2018, 7:28am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/12 "2018-08-22T07:28:07Z")
---
**Author:** ![pastechecker](https://avatars.discourse-cdn.com/v4/letter/p/0ea827/32.png) [@pastechecker](https://discuss.elastic.co/u/pastechecker)\ **Post date:** [August 30, 2018, 7:54am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/13 "2018-08-30T07:54:54Z")
Small update: With the loader.py I was able to load approximately 700GB (3,000,000,000,000 documents, json new line delimited with max line len not exceeding 400 characters). Average indexing speed used was 10,000 doc per batch with 16 threads. Hardware used: 2 ES nodes in a cluster connected by 1Gbps Ethernet card. Node configuration: 64 GB RAM ECC, Thread Ripper x1900 16 cores CPU, 4x2TB HDD configured in RAID0 using Debian 9.3 OS. There was 1 Gbps connection between the nodes. I am not doing aggressive queries on the cluster, just using it for storing data. At the moment I have: Indices: **44** Memory: **40.5 GB / 67.8 GB** Total Shards: **215** Unassigned Shards: **2** Documents: **3,249,292,622** Data: **1.0 TB** ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 5:27am UTC](https://discuss.elastic.co/t/loading-500-gb-of-data-3500000000-documents-to-es-cluster/137935/14 "2022-11-04T05:27:31Z")