# Reindexing 20TB document tips **URL:** \ **Category:** Elasticsearch\ **Created:** [June 29, 2019, 2:54pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140 "2019-06-29T14:54:48Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![zatom](https://avatars.discourse-cdn.com/v4/letter/z/f0a364/32.png) [@zatom](https://discuss.elastic.co/u/zatom)\ **Post date:** [June 29, 2019, 2:54pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/1 "2019-06-29T14:54:48Z")
Hi, Everytime I am trying to reindex this index with 20million documents(20TB) with reindex api, it stops in the half way. I even tried using sliced scroll(slicing) method to break it into multiple jobs and parallelized the reindexing, but stops after sometime. Sometimes, either the node goes down or the cluster's overall health goes bad while doing this. Is there another efficient way of reindexing such huge documents without knocking my cluster? Thank you ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2019, 4:27pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/2 "2019-06-29T16:27:19Z")
What does your reindx job look like? ---
**Author:** ![zatom](https://avatars.discourse-cdn.com/v4/letter/z/f0a364/32.png) [@zatom](https://discuss.elastic.co/u/zatom)\ **Post date:** [June 29, 2019, 5:53pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/3 "2019-06-29T17:53:30Z")
I am doing through kibana console. Following query worked until halfway through and knocked down of the node and stopped completely. My query looks like this: POST \_reindex?slices=20&timeout=60m&scroll=60m { "source":{ "index": "source\_index" }, "dest": { "index": "destination\_index", } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2019, 5:54pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/4 "2019-06-29T17:54:33Z")
Is this a single index? How many shards does it have? ---
**Author:** ![zatom](https://avatars.discourse-cdn.com/v4/letter/z/f0a364/32.png) [@zatom](https://discuss.elastic.co/u/zatom)\ **Post date:** [June 29, 2019, 5:55pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/5 "2019-06-29T17:55:55Z")
I had 20 shards. Yes, its going to single index ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2019, 5:56pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/6 "2019-06-29T17:56:53Z")
So 1TB per shard??? What is the specification of the cluster? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2019, 5:58pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/7 "2019-06-29T17:58:17Z")
What changes are you making for the destination index? ---
**Author:** ![zatom](https://avatars.discourse-cdn.com/v4/letter/z/f0a364/32.png) [@zatom](https://discuss.elastic.co/u/zatom)\ **Post date:** [June 29, 2019, 5:59pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/8 "2019-06-29T17:59:12Z")
yes. Is more shards the better? I saw with 20 shards, it was faster while it lasted. Later I tried on less shards, it was slower and stopped eventually ---
**Author:** ![zatom](https://avatars.discourse-cdn.com/v4/letter/z/f0a364/32.png) [@zatom](https://discuss.elastic.co/u/zatom)\ **Post date:** [June 29, 2019, 6:07pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/9 "2019-06-29T18:07:42Z")
@Christian_Dahlqvist I have another index with similar number of documents but with only 5 TB size. I am trying to reindex that also. I tried that with 20 shards. That failed too ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 29, 2019, 6:08pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/10 "2019-06-29T18:08:40Z")
I have never dealt with shards that large so will have to leave it to someone else I am afraid. ---
**Author:** ![iukea](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iukea/32/49083_2.png) [@iukea](https://discuss.elastic.co/u/iukea)\ **Post date:** [June 29, 2019, 9:10pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/11 "2019-06-29T21:10:08Z")
hold on @zatom > [@zatom](#): > > halfway through and knocked down of the node are you reindexing 20 TB of data on one node? Do you have a master node and a couple of data nodes? ---
**Author:** ![iukea](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iukea/32/49083_2.png) [@iukea](https://discuss.elastic.co/u/iukea)\ **Post date:** [June 29, 2019, 9:21pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/12 "2019-06-29T21:21:32Z")
I mean if you are doing it on one node I got mad respect for you. To better increase reindex speed (This will cause the server or servers to be maxed out at 100% utilization for a bit) I would do ``` POST _reindex { "size": 10000, "source": { "index": "oldBigAssIndex" }, "dest": { "index": "newBigassIndex" } } ``` By default, if the size is not set it only does 1,000 documents at a time the max size you can do is 10,000 without editing some properties. Reference [https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 30, 2019, 5:53am UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/13 "2019-06-30T05:53:59Z")
I'm not aware of any regular tests of creating such large shards, so you're a little off the beaten track here. I'd like to know more about how it is failing. Why is the node going down? Does it log anything about its failure? What are the mapping and settings for this index? ---
**Author:** ![zatom](https://avatars.discourse-cdn.com/v4/letter/z/f0a364/32.png) [@zatom](https://discuss.elastic.co/u/zatom)\ **Post date:** [July 1, 2019, 2:17pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/14 "2019-07-01T14:17:27Z")
I have 1 master node, 3 data nodes ---
**Author:** ![zatom](https://avatars.discourse-cdn.com/v4/letter/z/f0a364/32.png) [@zatom](https://discuss.elastic.co/u/zatom)\ **Post date:** [July 1, 2019, 2:21pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/15 "2019-07-01T14:21:08Z")
putting size outside the source will give me only that number of documents. I think you mean putting size inside the source. I will try that too ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 29, 2019, 2:21pm UTC](https://discuss.elastic.co/t/reindexing-20tb-document-tips/188140/16 "2019-07-29T14:21:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.