# Index rebuild taking much time **URL:** \ **Category:** Elasticsearch\ **Created:** [February 20, 2018, 5:26am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556 "2018-02-20T05:26:55Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![sateesh](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sateesh](https://discuss.elastic.co/u/sateesh)\ **Post date:** [February 20, 2018, 5:26am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/1 "2018-02-20T05:26:55Z")
Hi, I have only 170GB of Data, each time restarting data node, for rebuilding index taking to much time active percentage increased 18 to 19 it took almost 24 hr, what could be the issue. { "cluster\_name" : "elk-stack", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 8, "number\_of\_data\_nodes" : 3, "active\_primary\_shards" : 201, "active\_shards" : 402, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 1640, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 0, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 0, "active\_shards\_percent\_as\_number" : 19.686581782566112 Regards sateesh ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 20, 2018, 6:22am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/2 "2018-02-20T06:22:40Z")
If you only have 170GB of data, why do you have so many shards? ---
**Author:** ![sateesh](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sateesh](https://discuss.elastic.co/u/sateesh)\ **Post date:** [February 20, 2018, 6:55am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/3 "2018-02-20T06:55:43Z")
Not sure what went wrong, i have deleted the index even after that so many shards is there, how to clear this one. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 20, 2018, 7:28am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/4 "2018-02-20T07:28:42Z")
Are you using time based indices? How many indices are you creating per day? Are you using the default settings (5 primary shards, 1 replica)? ---
**Author:** ![sateesh](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sateesh](https://discuss.elastic.co/u/sateesh)\ **Post date:** [February 20, 2018, 8:12am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/5 "2018-02-20T08:12:32Z")
Yes using time stamp filter { if [fields][log\_type] == "test\_access" { grok { match =\> { "message" =\> "%{GREEDYDATA}" } add\_tag =\> ["test\_access"] remove\_tag =\> ["\_grokparsefailure", "beats\_input\_codec\_plain\_applied"] } date { match =\> ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"] } } } output { if [fields][log\_type] == "test\_access" { elasticsearch { hosts =\> ["x.x.x.x:9200"] index =\> "test-%{+YYYY.MM.dd}" } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 20, 2018, 8:24am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/6 "2018-02-20T08:24:24Z")
If you only have one time-based index, you can probably switch to using a monthly index instead given the volumes. You should also be able to reduce the number of primary shards through an [index template](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-templates.html). > [@sateesh](#): > > match =\> { "message" =\> "%{GREEDYDATA}" } What is the point of this grok expression? ---
**Author:** ![sateesh](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sateesh](https://discuss.elastic.co/u/sateesh)\ **Post date:** [February 21, 2018, 12:08pm UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/7 "2018-02-21T12:08:32Z")
One of the data node shows the below error, looks like this is the causing issue. error logs-[SERVICE\_UNAVAILABLE/2/no master This is \_cluster/\_status?pretty' ---status { "error" : { "root\_cause" : [ { "type" : "illegal\_argument\_exception", "reason" : "No feature for name [\_status]" } ], "type" : "illegal\_argument\_exception", "reason" : "No feature for name [\_status]" }, "status" : 400 } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [February 21, 2018, 12:16pm UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/8 "2018-02-21T12:16:18Z")
Can you provide the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/6.2/cluster-stats.html)? How much heap do you data nodes have assigned? ---
**Author:** ![sateesh](https://avatars.discourse-cdn.com/v4/letter/s/e274bd/32.png) [@sateesh](https://discuss.elastic.co/u/sateesh)\ **Post date:** [February 22, 2018, 6:28am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/9 "2018-02-22T06:28:06Z")
> [@Christian\_Dahlqvist](#): > > cluster stats API? { "timestamp" : 1519280649176, "cluster\_name" : "elk-elastic", "status" : "red", "indices" : { "count" : 43, "shards" : { "total" : 422, "primaries" : 211, "replication" : 1.0, "index" : { "shards" : { "min" : 2, "max" : 10, "avg" : 9.813953488372093 }, "primaries" : { "min" : 1, "max" : 5, "avg" : 4.906976744186046 }, "replication" : { "min" : 1.0, "max" : 1.0, "avg" : 1.0 } } }, "docs" : { "count" : 855052872, "deleted" : 2 }, "store" : { "size" : "652.1gb", "size\_in\_bytes" : 700226555319, "throttle\_time" : "0s", "throttle\_time\_in\_millis" : 0 }, "fielddata" : { "memory\_size" : "0b", "memory\_size\_in\_bytes" : 0, "evictions" : 0 }, "query\_cache" : { "memory\_size" : "31.3kb", "memory\_size\_in\_bytes" : 32096, "total\_count" : 1110181, "hit\_count" : 0, "miss\_count" : 1110181, "cache\_size" : 57, "cache\_count" : 128, "evictions" : 71 }, "completion" : { "size" : "0b", "size\_in\_bytes" : 0 }, "segments" : { "count" : 6015, "memory" : "2.1gb", "memory\_in\_bytes" : 2302303124, "terms\_memory" : "1.9gb", "terms\_memory\_in\_bytes" : 2046090264, "stored\_fields\_memory" : "237.8mb", "stored\_fields\_memory\_in\_bytes" : 249426656, "term\_vectors\_memory" : "0b", "term\_vectors\_memory\_in\_bytes" : 0, "norms\_memory" : "4.5mb", "norms\_memory\_in\_bytes" : 4792384, "doc\_values\_memory" : "1.9mb", "doc\_values\_memory\_in\_bytes" : 1993820, "index\_writer\_memory" : "9.5mb", "index\_writer\_memory\_in\_bytes" : 10044416, "index\_writer\_max\_memory" : "9.5gb", "index\_writer\_max\_memory\_in\_bytes" : 10292884036, "version\_map\_memory" : "969.6kb", "version\_map\_memory\_in\_bytes" : 992940, "fixed\_bit\_set" : "0b", "fixed\_bit\_set\_memory\_in\_bytes" : 0 }, "percolate" : { "total" : 0, "time" : "0s", "time\_in\_millis" : 0, "current" : 0, "memory\_size\_in\_bytes" : -1, "memory\_size" : "-1b", "queries" : 0 } }, "nodes" : { "count" : { "total" : 8, "master\_only" : 3, "data\_only" : 3, "master\_data" : 0, "client" : 0 }, "versions" : ["2.4.6"], "os" : { "available\_processors" : 32, "allocated\_processors" : 32, "mem" : { "total" : "53.4gb", "total\_in\_bytes" : 57378734080 }, "names" : [ { "name" : "Linux", "count" : 8 } ] }, "process" : { "cpu" : { "percent" : 9 }, "open\_file\_descriptors" : { "min" : 343, "max" : 3439, "avg" : 1485 } }, "jvm" : { "max\_uptime" : "16.7d", "max\_uptime\_in\_millis" : 1451058144, "versions" : [ { "version" : "1.8.0\_151", "vm\_name" : "OpenJDK 64-Bit Server VM", "vm\_version" : "25.151-b12", "vm\_vendor" : "Oracle Corporation", "count" : 8 } ], "mem" : { "heap\_used" : "8.4gb", "heap\_used\_in\_bytes" : 9042331416, "heap\_max" : "31.7gb", "heap\_max\_in\_bytes" : 34080817152 }, "threads" : 501 }, "fs" : { "total" : "4.6tb", "total\_in\_bytes" : 5073091891200, "free" : "3.9tb", "free\_in\_bytes" : 4366264889344, "available" : "3.7tb", "available\_in\_bytes" : 4150526734336 }, "plugins" : [ { "name" : "head", "version" : "master", "description" : "head - A web front end for an Elasticsearch cluster", "url" : "/\_plugin/head/", "jvm" : false, "site" : true }, { "name" : "elastic-cloud", "version" : "2.4.6", "description" : "The Amazon Web Service (AWS) Cloud plugin allows to use AWS API for the unicast discovery mechanism and add S3 repositories.", "jvm" : true, "classname" : "org.elasticsearch.plugin.cloud.aws.CloudAwsPlugin", "isolated" : true, "site" : false }, { "name" : "analysis-kuromoji", "version" : "2.4.6", "description" : "The Japanese (kuromoji) Analysis plugin integrates Lucene kuromoji analysis module into elasticsearch.", "jvm" : true, "classname" : "org.elasticsearch.plugin.analysis.kuromoji.AnalysisKuromojiPlugin", "isolated" : true, "site" : false }, { "name" : "analysis-icu", "version" : "2.4.6", "description" : "The ICU Analysis plugin integrates Lucene ICU module into elasticsearch, adding ICU relates analysis components.", "jvm" : true, "classname" : "org.elasticsearch.plugin.analysis.icu.AnalysisICUPlugin", "isolated" : true, "site" : false } ] } } ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 22, 2018, 6:28am UTC](https://discuss.elastic.co/t/index-rebuild-taking-much-time/120556/10 "2018-03-22T06:28:20Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.