# Slow Index create **URL:** \ **Category:** Elasticsearch\ **Created:** [October 23, 2019, 12:02pm UTC](https://discuss.elastic.co/t/slow-index-create/204862 "2019-10-23T12:02:23Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![gintek](https://avatars.discourse-cdn.com/v4/letter/g/f05b48/32.png) [@gintek](https://discuss.elastic.co/u/gintek)\ **Post date:** [October 23, 2019, 12:02pm UTC](https://discuss.elastic.co/t/slow-index-create/204862/1 "2019-10-23T12:02:23Z")
Hi, We are struggling with ES performance issue, we have daily indexes and minute or two after midnight our cluster is unavailable. Logstash indexer reports time out, after around 10 min everything back to normal. we try few different strategies, create indexer with 0 replica and replica during the day. In pick we creates 150 indexes. After investigation we discovered index create time is around 8 sec, even after high pick still it's 8sec. We believe it's root cause. My questions: 1. How to speed up index creation ? 2. Does it make sense have dedicated servers and VMs ? Does cluster speed depends on slowest node ? Our cluster: 3 masters, 3 hot nodes (dedicated bare-metal serves) 6 cold nodes VMs, 1 ingest node. We have 20k shards, 15k on cold nodes. "version" : { "number" : "6.4.2", "build\_flavor" : "default", "build\_type" : "rpm", "build\_hash" : "04711c2", "build\_date" : "2018-09-26T13:34:09.098244Z", "build\_snapshot" : false, "lucene\_version" : "7.4.0", Thanks for any suggestions. ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [October 24, 2019, 9:19am UTC](https://discuss.elastic.co/t/slow-index-create/204862/2 "2019-10-24T09:19:33Z")
You can have a look at [Rollover API](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/indices-rollover-index.html) or write a custom job to pre-create indices in advance (the idea is to spread mass index creation over a longer period of time). HTH, Lukáš ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 24, 2019, 9:25am UTC](https://discuss.elastic.co/t/slow-index-create/204862/3 "2019-10-24T09:25:26Z")
> [@gintek](#): > > We have 20k shards, 15k on cold nodes. You have far too many shards, and this can indeed result in performance issues. Here is an article with some more detailed guidance: > **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** ---
**Author:** ![gintek](https://avatars.discourse-cdn.com/v4/letter/g/f05b48/32.png) [@gintek](https://discuss.elastic.co/u/gintek)\ **Post date:** [October 24, 2019, 10:06am UTC](https://discuss.elastic.co/t/slow-index-create/204862/4 "2019-10-24T10:06:27Z")
Thanks for reply, How many shards I can have on data nodes ? Psychical servers 256G RAM, 72CPU, fast disks. I'm a little bit confused, in many places recommended numbers shards is 1K per node, but what if node is really fast server ? What is better way to scale ElasticSearch vertical or horizontal ? May be it doesn't make sense pay for really fast servers, better invest in many small instances. I'm really curios about your opinion. Thanks. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 24, 2019, 10:28am UTC](https://discuss.elastic.co/t/slow-index-create/204862/5 "2019-10-24T10:28:31Z")
The recommendation is significantly less than 1000 shards per node. The guideline in the article I shared is 20 shards per GB of heap, and you should have less than 32GB of heap, so that puts the recommended limit at less than 640 shards. It also recommends a target shard size of 20-40GB, which adds up to 12-25TB per node. If you have more data than that then either target larger shards or else scale out horizontally. ---
**Author:** ![gintek](https://avatars.discourse-cdn.com/v4/letter/g/f05b48/32.png) [@gintek](https://discuss.elastic.co/u/gintek)\ **Post date:** [October 24, 2019, 11:41am UTC](https://discuss.elastic.co/t/slow-index-create/204862/6 "2019-10-24T11:41:25Z")
@DavidTurner thanks for explanation. Please correct me if I'm wrong but I've feeling it's doesn't make sense buy expensive servers ( 256G RAM, 72CPU ) because of ES limit, heap should no more 32G. Seems optimal servers is around 64G RAM and scale out horizontally instead invest in expensive fast bare-metal server. Right ? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [October 24, 2019, 11:47am UTC](https://discuss.elastic.co/t/slow-index-create/204862/7 "2019-10-24T11:47:34Z")
The optimal hardware for your cluster very much depends on your needs and you can only really determine the best setup with careful benchmarking of a realistic workload. There are definitely workloads that benefit from 256GB or more of RAM. Elasticsearch will use any spare RAM for the filesystem cache, which can be many times larger than the JVM heap. ---
**Author:** ![gintek](https://avatars.discourse-cdn.com/v4/letter/g/f05b48/32.png) [@gintek](https://discuss.elastic.co/u/gintek)\ **Post date:** [October 24, 2019, 12:47pm UTC](https://discuss.elastic.co/t/slow-index-create/204862/8 "2019-10-24T12:47:26Z")
@DavidTurner and @Lukas_Vlcek1 thank you a lot ! I'm going back to my cluster, seems I need review our assumptions. You can close this it. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 21, 2019, 12:47pm UTC](https://discuss.elastic.co/t/slow-index-create/204862/9 "2019-11-21T12:47:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.