# Elastic uses hardware minimally while reindexing **URL:** \ **Category:** Elasticsearch\ **Created:** [January 16, 2018, 9:51pm UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793 "2018-01-16T21:51:35Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![seb\_ma](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@seb\_ma](https://discuss.elastic.co/u/seb_ma)\ **Post date:** [January 16, 2018, 9:51pm UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793/1 "2018-01-16T21:51:35Z")
Hello, our Setup: 4 Server, each with 2 cpus (each 24 vCore) 256 GB Ram, 2 SSDs (each 120GB) Raid0, 10Gbit interconnect. On each Server we have 4 ES 6.1.1 Docker-Container running, created with docker compose. Each ES-Node has 29GB Heap. Docker has no limits to Hw. Now we wanted to reindex an index with 120GB of Data (16 shards, 1 replica). No other tasks are running on the servers or nodes. The cluster can use the complete Hardware exclusively for reindexing. After we started the reindexing task, we were wondering why the cluster is not realy using the hardware. Only some vcores on the servers were running with around 10%, the SSD I/O were between 0 and 20% and the Lan interconnect has been used minimally. Finally it took 3 hours for the task. Can sombody tell us, why the hardware is not really used while reindexing? Thank You! ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [January 19, 2018, 1:17pm UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793/2 "2018-01-19T13:17:03Z")
did you look into [slicing](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-slice) for you reindex task? ---
**Author:** ![seb\_ma](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@seb\_ma](https://discuss.elastic.co/u/seb_ma)\ **Post date:** [January 19, 2018, 4:28pm UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793/3 "2018-01-19T16:28:56Z")
Hi Simon, yes, we tryed the slicing. But after finishing the reindexing, the new index has less documents then the original one. So, we were loosing data and didn't try slicing again. We set the vm.max\_map\_count to 262144 as recommended. Is it maybe to small for 4 docker machines? ---
**Author:** ![s1monw](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/s1monw/32/3637_2.png) [@s1monw](https://discuss.elastic.co/u/s1monw)\ **Post date:** [January 22, 2018, 9:23am UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793/4 "2018-01-22T09:23:15Z")
> [@seb\_ma](#): > > yes, we tryed the slicing. But after finishing the reindexing, the new index has less documents then the original one. So, we were loosing data and didn't try slicing again. wait, what? I mean that sounds like a terrible bug to me. What version of elasticsearch are you using and do you keep on writing to the index you are reindexing from or do you reindex into the same index? /cc @jimczi @nik9000 ---
**Author:** ![seb\_ma](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@seb\_ma](https://discuss.elastic.co/u/seb_ma)\ **Post date:** [January 22, 2018, 11:32am UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793/5 "2018-01-22T11:32:25Z")
We are using your offizial Docker Image for 6.1.1. The index was "finished". We don't keep writing to the index any more. We also created a new index in which we were reindexing. ---
**Author:** ![jimczi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jimczi/32/47985_2.png) [@jimczi](https://discuss.elastic.co/u/jimczi)\ **Post date:** [January 22, 2018, 1:46pm UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793/6 "2018-01-22T13:46:12Z")
Can you share the complete request that you used to reindex with slice ? Did you use automatic slicing: [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-automatic-slice](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-automatic-slice) or manual: [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-manual-slice](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html#docs-reindex-manual-slice) ? Did you check for failures in the response of the reindex task ? Slicing is the preferred way to parallelize a single reindex task so it shouldn't miss any document. Are you able to retry the operation and record the response (the task response) if not already done ? ---
**Author:** ![seb\_ma](https://avatars.discourse-cdn.com/v4/letter/s/898d66/32.png) [@seb\_ma](https://discuss.elastic.co/u/seb_ma)\ **Post date:** [February 5, 2018, 12:54pm UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793/7 "2018-02-05T12:54:55Z")
``` Indices created by: PUT index2 { "settings": { "number_of_replicas": 0, "number_of_shards": 20, "refresh_interval" : "300s" } } PUT index3 { "settings": { "number_of_replicas": 0, "number_of_shards": 20, "refresh_interval" : "300s" } } Reindex with slice: POST _reindex { "source": { "index": "index1", "slice": { "id": 0, "max": 2 } }, "dest": { "index": "index2" } } POST _reindex { "source": { "index": "index1", "slice": { "id": 1, "max": 2 } }, "dest": { "index": "index2" } } POST _reindex { "source": { "index": "index1", "slice": { "id": 0, "max": 2 } }, "dest": { "index": "index3" } } POST _reindex { "source": { "index": "index1", "slice": { "id": 1, "max": 2 } }, "dest": { "index": "index3" } } ``` We used this commands for creating the Index and manual slicing. We were not able to reproduce the problem again. we tryed it with autmatic slicing and it worked fine. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 5, 2018, 12:55pm UTC](https://discuss.elastic.co/t/elastic-uses-hardware-minimally-while-reindexing/115793/8 "2018-03-05T12:55:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.