# Full Index Rebuild with 5M documents on Elastic search 7.6 takes 1 week to complete **URL:** \ **Category:** Elasticsearch\ **Created:** [November 17, 2020, 6:21am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624 "2020-11-17T06:21:17Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 6:21am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/1 "2020-11-17T06:21:17Z")
Hey There Hope all are doing well Please have a look in to the below issue and provide your valuable suggestion on this. Currently Elastic 7.6 is running on cluster with 3 nodes and it's having 5M records in it. While performing the Full rebuild of index is taking huge amount of time. But other server metrics like CPU and GC is within the limit And similarly I had tried ES re index process which is also time consuming. Applied few workarounds like by setting Refresh interval to -1 and Replica to 0 but it still takes very long time. I have almost 168 shards but the segment count is lil more which is showing 800+. Kindly help and suggest to resolve this issue. Any leads would be much appreciated. Thanks ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 17, 2020, 6:52am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/2 "2020-11-17T06:52:05Z")
How large, in GB, is the index? ---
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 7:23am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/3 "2020-11-17T07:23:24Z")
Thanks Mark for your response. Max index size has used during 5M testing was up to 30GB. Please find below es1\_development-authoring 2 p STARTED 249395 17.9gb 100.105.8.84 node-3 es1\_development-authoring 2 r STARTED 248955 18.5gb 100.105.8.74 node-1 es1\_development-authoring 1 p STARTED 247935 18.2gb 100.105.8.75 node-2 es1\_development-authoring 1 r STARTED 245468 19.1gb 100.105.8.74 node-1 es1\_development-authoring 3 p STARTED 247987 18.5gb 100.105.8.75 node-2 es1\_development-authoring 3 r STARTED 246748 18.4gb 100.105.8.74 node-1 es1\_development-authoring 4 p STARTED 248449 18.7gb 100.105.8.84 node-3 es1\_development-authoring 4 r STARTED 246067 17.9gb 100.105.8.75 node-2 es1\_development-authoring 0 r STARTED 246480 20.6gb 100.105.8.84 node-3 es1\_development-authoring 0 p STARTED 247545 19gb 100.105.8.74 node-1 es1\_development-review 2 r STARTED 247798 18gb 100.105.8.84 node-3 es1\_development-review 2 p STARTED 245559 23gb 100.105.8.74 node-1 es1\_development-review 1 p STARTED 246717 18gb 100.105.8.75 node-2 es1\_development-review 1 r STARTED 244555 17.9gb 100.105.8.74 node-1 es1\_development-review 3 r STARTED 246507 17.8gb 100.105.8.75 node-2 es1\_development-review 3 p STARTED 244215 17.9gb 100.105.8.74 node-1 es1\_development-review 4 r STARTED 246592 20.4gb 100.105.8.84 node-3 es1\_development-review 4 p STARTED 246592 18gb 100.105.8.75 node-2 es1\_development-review 0 r STARTED 246243 17.8gb 100.105.8.84 node-3 es1\_development-review 0 p STARTED 244675 22.9gb 100.105.8.74 node-1 es1\_development-published 2 r STARTED 227358 25gb 100.105.8.75 node-2 es1\_development-published 2 p STARTED 227358 30.4gb 100.105.8.74 node-1 es1\_development-published 1 p STARTED 227314 31.5gb 100.105.8.75 node-2 es1\_development-published 1 r STARTED 227314 30.9gb 100.105.8.74 node-1 es1\_development-published 3 p STARTED 227912 29gb 100.105.8.84 node-3 es1\_development-published 3 r STARTED 227912 31.7gb 100.105.8.74 node-1 es1\_development-published 4 r STARTED 227064 24.9gb 100.105.8.84 node-3 es1\_development-published 4 p STARTED 227064 28.1gb 100.105.8.75 node-2 es1\_development-published 0 p STARTED 227586 27.4gb 100.105.8.84 node-3 es1\_development-published 0 r STARTED 227586 28.4gb 100.105.8.74 node-1 es1\_development-approval 2 p STARTED 241330 10.7gb 100.105.8.84 node-3 es1\_development-approval 2 r STARTED 241330 10.8gb 100.105.8.74 node-1 es1\_development-approval 1 p STARTED 242573 10.6gb 100.105.8.84 node-3 es1\_development-approval 1 r STARTED 242573 10.7gb 100.105.8.74 node-1 es1\_development-approval 3 r STARTED 242309 10.6gb 100.105.8.84 node-3 es1\_development-approval 3 p STARTED 242309 10.7gb 100.105.8.75 node-2 es1\_development-approval 4 p STARTED 242142 10.8gb 100.105.8.84 node-3 es1\_development-approval 4 r STARTED 242142 10.8gb 100.105.8.75 node-2 es1\_development-approval 0 p STARTED 242079 10.7gb 100.105.8.75 node-2 es1\_development-approval 0 r STARTED 242079 10.6gb 100.105.8.74 node-1 ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 17, 2020, 7:24am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/4 "2020-11-17T07:24:14Z")
Ok, do you have 168 shards for this one, 30GB index? ---
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 7:24am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/5 "2020-11-17T07:24:48Z")
Total it shows 168 for all 3 nodes in Kibana. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 17, 2020, 7:25am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/6 "2020-11-17T07:25:41Z")
Yeah, but how many for this single index you are reindexing? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 17, 2020, 7:33am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/7 "2020-11-17T07:33:59Z")
What you have listed are the individual shards, so it looks like the indices you have shown all have 5 primary shards and most of them seem to be over 100GB in size. It would help if you could provide some details about the hardware and configuration of your cluster as well as details about exactly how you are reindexing. ---
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 7:44am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/8 "2020-11-17T07:44:15Z")
Yes It uses default config that is 5 shards and all total Kibana shows 31 indices Server Configuration :- 8 core ocpu with 54 GB RAM and where 26 GB assigned to heap on each node So far it has used total storage on 3 nodes almost 1000 GB Please find below index setting { "settings": { "index": { "number\_of\_shards": "5", "provided\_name": "es1\_development-authoring", "creation\_date": "1605507707367", "analysis": { "filter": { "stemmer\_filter": { "type": "stemmer", "language": "english" }, "word\_delimiter\_filter": { "split\_on\_numerics": "false", "split\_on\_case\_change": "false", "generate\_word\_parts": "true", "type": "word\_delimiter", "generate\_number\_parts": "true" } }, "char\_filter": { "html\_strip\_char\_filter": { "type": "html\_strip" } }, "normalizer": { "lowercase\_normalizer": { "filter": [ "lowercase" ], "type": "custom" } }, "analyzer": { "stem\_analyzer": { "filter": [ "lowercase", "word\_delimiter\_filter", "stemmer\_filter" ], "char\_filter": [ "html\_strip\_char\_filter" ], "tokenizer": "whitespace" }, "word\_delimiter\_analyzer": { "filter": [ "lowercase", "word\_delimiter\_filter" ], "char\_filter": [ "html\_strip\_char\_filter" ], "type": "custom", "tokenizer": "whitespace" } } }, "number\_of\_replicas": "1", "uuid": "cB3GcK07RAmlmkdieUFv7A", "version": { "created": "7060099" } } }, "defaults": { "index": { "flush\_after\_merge": "512mb", "final\_pipeline": "\_none", "max\_inner\_result\_window": "100", "unassigned": { "node\_left": { "delayed\_timeout": "1m" } }, "max\_terms\_count": "65536", "lifecycle": { "name": "", "parse\_origination\_date": "false", "indexing\_complete": "false", "rollover\_alias": "", "origination\_date": "-1" }, "routing\_partition\_size": "1", "force\_memory\_term\_dictionary": "false", "max\_docvalue\_fields\_search": "100", "merge": { "scheduler": { "max\_thread\_count": "4", "auto\_throttle": "true", "max\_merge\_count": "9" }, "policy": { "reclaim\_deletes\_weight": "2.0", "floor\_segment": "2mb", "max\_merge\_at\_once\_explicit": "30", "max\_merge\_at\_once": "10", "max\_merged\_segment": "5gb", "expunge\_deletes\_allowed": "10.0", "segments\_per\_tier": "10.0", "deletes\_pct\_allowed": "33.0" } }, "max\_refresh\_listeners": "1000", "max\_regex\_length": "1000", "load\_fixed\_bitset\_filters\_eagerly": "true", "number\_of\_routing\_shards": "1", "write": { "wait\_for\_active\_shards": "1" }, "verified\_before\_close": "false", "mapping": { "coerce": "false", "nested\_fields": { "limit": "50" }, "depth": { "limit": "20" }, "field\_name\_length": { "limit": "9223372036854775807" }, "total\_fields": { "limit": "1000" }, "nested\_objects": { "limit": "10000" }, "ignore\_malformed": "false" }, "source\_only": "false", "soft\_deletes": { "enabled": "false", "retention": { "operations": "0" }, "retention\_lease": { "period": "12h" } }, "max\_script\_fields": "32", "query": { "default\_field": [ "\*" ], "parse": { "allow\_unmapped\_fields": "true" } }, "format": "0", "frozen": "false", "sort": { "missing": , "mode": , "field": , "order": }, "priority": "1", "codec": "default", "max\_rescore\_window": "10000", "max\_adjacency\_matrix\_filters": "100", "analyze": { "max\_token\_count": "10000" }, "gc\_deletes": "60s", "optimize\_auto\_generated\_id": "true", "max\_ngram\_diff": "1", "translog": { "generation\_threshold\_size": "64mb", "flush\_threshold\_size": "512mb", "sync\_interval": "5s", "retention": { "size": "512MB", "age": "12h" }, "durability": "REQUEST" }, "auto\_expand\_replicas": "false", "mapper": { "dynamic": "true" }, "requests": { "cache": { "enable": "true" } }, "data\_path": "", "highlight": { "max\_analyzed\_offset": "1000000" }, "routing": { "rebalance": { "enable": "all" }, "allocation": { "enable": "all", "total\_shards\_per\_node": "-1" } }, "search": { "slowlog": { "level": "TRACE", "threshold": { "fetch": { "warn": "-1", "trace": "-1", "debug": "-1", "info": "-1" }, "query": { "warn": "-1", "trace": "-1", "debug": "-1", "info": "-1" } } }, "idle": { "after": "30s" }, "throttled": "false" }, "fielddata": { "cache": "node" }, "default\_pipeline": "\_none", "max\_slices\_per\_scroll": "1024", "shard": { "check\_on\_startup": "false" }, "xpack": { "watcher": { "template": { "version": "" } }, "version": "", "ccr": { "following\_index": "false" } }, "percolator": { "map\_unmapped\_fields\_as\_text": "false" }, "allocation": { "max\_retries": "5" }, "refresh\_interval": "1s", "indexing": { "slowlog": { "reformat": "true", "threshold": { "index": { "warn": "-1", "trace": "-1", "debug": "-1", "info": "-1" } }, "source": "1000", "level": "TRACE" } }, "compound\_format": "0.1", "blocks": { "metadata": "false", "read": "false", "read\_only\_allow\_delete": "false", "read\_only": "false", "write": "false" }, "max\_result\_window": "10000", "store": { "stats\_refresh\_interval": "10s", "type": "", "fs": { "fs\_lock": "native" }, "preload": }, "queries": { "cache": { "enabled": "true" } }, "warmer": { "enabled": "true" }, "max\_shingle\_diff": "3", "query\_string": { "lenient": "false" } } } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 17, 2020, 7:49am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/9 "2020-11-17T07:49:36Z")
What type of storage are you using? Exactly how are you reindexing the data? If you are using the reindex API, are you [slicing](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/docs-reindex.html#docs-reindex-slice) to improve concurrency? It may also help if you tell us a bit about your data as the documents look quite large. ---
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 8:18am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/10 "2020-11-17T08:18:50Z")
Hey Christian Please find below information I am using block volume storage and each node has BV as 1T in each Local SSD is almost 250GB on each node. Yes it uses Reindex API of Elasticsearch and it is configured for auto slicing. ES will decide on the number of slices depends on the volume and shards. As part of data, I have seeded .pdf(56KB),.png(10KB), .docx(10k to 100k) and .txt (10K size) So , For 5M the total size is almost 300-400 GB. Thanks ---
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 8:22am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/11 "2020-11-17T08:22:52Z")
Currently in my system, for Re indexing it takes 18-19 hrs to complete. Can we still improve ? But where as full index rebuild is very expensive and taking 1 week of time to complete ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 17, 2020, 8:33am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/12 "2020-11-17T08:33:48Z")
> [@nimohara](#): > > I am using block volume storage and each node has BV as 1T in each In assume this is where data is stored. What does disk I/O and iowait look like for these volumes during reindexing? Are you storing binary content ion base64 encoded for in the index? ---
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 8:53am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/13 "2020-11-17T08:53:03Z")
Yes you are right. This is where the data is stored. And I dint find any disk i/o wait issue during the process. In fact the cpu consumption was almost 30-40%. A And exactly not very sure about internally how it stores in indexes? ---
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 8:57am UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/14 "2020-11-17T08:57:46Z")
In the current environment setup, I have already validated small amount of data set that 270k records and during that time it was faster ES reindex and Full Index rebuild. But when I increased upto 5M , slowness observed. ---
**Author:** ![nimohara](https://avatars.discourse-cdn.com/v4/letter/n/ad7895/32.png) [@nimohara](https://discuss.elastic.co/u/nimohara)\ **Post date:** [November 17, 2020, 2:46pm UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/15 "2020-11-17T14:46:50Z")
Hi I think yes based on below statement Records/documents are stored in Table segment and if there are any index created on the table. rowid , Column value would be stored in index segment.A logical rowid is a base64-encoded representation of the table primary key ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 15, 2020, 2:47pm UTC](https://discuss.elastic.co/t/full-index-rebuild-with-5m-documents-on-elastic-search-7-6-takes-1-week-to-complete/255624/16 "2020-12-15T14:47:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.