# What is the best mix of index and shards on Elastic Cloud **URL:** \ **Category:** Elasticsearch\ **Created:** [June 24, 2021, 9:03am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896 "2021-06-24T09:03:55Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 24, 2021, 9:03am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/1 "2021-06-24T09:03:55Z")
As the title says, we are using Elastic Cloud with 3 Elastic Instances. We are currently having 338 shards: ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/c/0/c0fe296d9197742eb585c1bfede637ba2f67ed17.png) but just having one index: ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1c2593d0cf4a9f334bfdbe888d36340261f12124.png) Is this a proper way? Or should I create one index for each day maybe? (btw before we migrated to Cloud we had one index per month) What is the best practice for Cloud? I don't want to struggle later because a huge index is making the system very slow because freezing is not possible for instance. So give me any tips you have on this. Thanks. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 27, 2021, 10:46pm UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/2 "2021-06-27T22:46:33Z")
This is using [ILM](https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html), which you can tell by the counter on the end of the date in the index name. This is the default of Beats these days and what you should be using. What version are you running? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 28, 2021, 12:27am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/3 "2021-06-28T00:27:12Z")
@kwoxer You might want to take a look at [this](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) article about shard sizing. First Glance that is a lot of shards for a cluster of that size we typically recommend 10-20 shards per 1 GB of Heap. Your total cluster has ~1.5GB of heap (50% of 3GB Total) so you should only have ~30 Shards Total you have 10x that, that will negatively affect your performance. It's possible this is contributing to your issue in your other thread. ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 28, 2021, 3:54am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/4 "2021-06-28T03:54:42Z")
I understand. So I try to reduce shards by 10x and having a look again on the performance issues I currently have. ---
**Author:** ![ropc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ropc/32/47022_2.png) [@ropc](https://discuss.elastic.co/u/ropc)\ **Post date:** [June 28, 2021, 4:06am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/5 "2021-06-28T04:06:38Z")
In addition to what my awesome colleagues shared, you can also take look at: - [Aim for shard sizes between 10GB and 50GB](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#shard-size-recommendation) - [Aim for 20 shards or fewer per GB of heap memory](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#shard-count-recommendation) - [Fix an oversharded cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html#fix-an-oversharded-cluster) You also have the possibility to scale-up your deployment if more resources are needed (c.f [Do you know when to scale?](https://www.elastic.co/guide/en/cloud/current/ec-planning.html#ec-workloads)). ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 28, 2021, 4:09am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/6 "2021-06-28T04:09:17Z")
Currently I am getting this message kind of everywhere ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/6/4/6416b852b157ed29a692a27b3a0a4b6acf3c5364.png) Could you tell me the query to reduce the number of shards? ---
**Author:** ![ropc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ropc/32/47022_2.png) [@ropc](https://discuss.elastic.co/u/ropc)\ **Post date:** [June 28, 2021, 4:17am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/7 "2021-06-28T04:17:43Z")
Your deployment is currently running on JVM heap usage, hence the parent circuit breaker exception observed. A couple of options here: 1. If you have unwanted / old indices that you are not using anymore, you can use the [Delete API](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-delete.html) to delete these indices. Each index has 1 or many shards (depending on your index settings). Deleting indices can be done for example via the Kibana Console or via the API Console (c.f [Access the Elasticsearch API console](https://www.elastic.co/guide/en/cloud/current/ec-api-console.html)). 2. Scale-up your deployment to allocate more resources. ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 28, 2021, 5:21am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/8 "2021-06-28T05:21:55Z")
I changed the number of shards in the index management. Is this really so heavy so that now Kibana is not available anymore? ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/d/a/da542e36655aff0e1804a96219fdf9a7f2323b21.png) ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/6/1/612295c869c740a913960e852ab4e3847b6a13df.png) ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/1/1/112d03b99f6d13b3e027cfef9adcad4cb3708933.png) ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/a/b/ab48364df41309890271dc45d7595aa2049c588e.png) How long does this take approx? Is there something I can do? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 28, 2021, 6:24am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/9 "2021-06-28T06:24:15Z")
There is only a couple very special command (shrink/split) to change the number of primary shards or combine indices. Neither of which is available through Kibana The number of primary shards for an index is set at create time. what exactly did you do? ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 28, 2021, 6:27am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/10 "2021-06-28T06:27:07Z")
I cannot show you. As Kibana is down. I was in index **Lifecycle management** of the Index and edited the **Hot Phase**. There I enables the **Shards** section and set the number to **30 shards** and saved. Now getting those errors... Btw now I get this message ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac42e5917a37d0209bcd824de28d98c90118c8f.png) But clearing cache does not help. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 28, 2021, 6:34am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/11 "2021-06-28T06:34:45Z")
ILM is not really the correct way to attempt to fix that. And you don't set the number of shards in ILM You set the index size before rollover so I am not sure what You really did. If I were you I'd open a support ticket. Once you get the cluster up and running you really should have left the default settings for the ILM an index templates which are The defaults recommended for time series data are: One primary shard One replica shard And in the hot phase of ILM 50 GB per index for rollover However your cluster is tiny, Since you only have 30 gigabytes storage of disc on each node obviously 50 GB shards won't work You can make each index size 2GB or so. That would give you 15 or so indexes / shards per node. So depending on what your actual data ingest and retention requirements are you may need to adjust the capacity / the size of your cluster. ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 28, 2021, 8:34am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/12 "2021-06-28T08:34:36Z")
> [@stephenb](#): > > wever your cluster is tiny, > > Since you only have 30 gigabytes storage of disc on each node obviously 50 GB shards And what should be the Maximum primary shard size? These settings are fine? ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/f/b/fb671dbd38ce82579176cb6254a08ca91edac664.png) ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 28, 2021, 8:42am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/13 "2021-06-28T08:42:08Z")
And where do I see where these shards are located or coming from? ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/7/0/7061fc8d8ed1b2b3aaa6d7401cfbdc306f929d5c.png) ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 28, 2021, 2:21pm UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/14 "2021-06-28T14:21:38Z")
Hi @kwoxer So you got the cluster green again? Did you actually look at / read all the great docs that @ropc and I shared? There was a lot of great info in those. IF you left the defaults for number of Primary and Replica shards to be 1 and 1 respectively then I would put that setting in ILM to 2 GB. BTW if you changed the number of Primary shards in your mapping or index template you should set it back to 1 or take it out. If you do not set it 1 is the default. Again your cluster is on the very small scale so these numbers are all a bit skewed. With respect to the shards, where they are located they should be pretty much evenly distributed across the 3 nodes. I would not get too concerned about where they are You can go to Kibana -\> Dev Tools and run `GET _cat/shards/?v` To see where they are . I highly recommend taking a look at the docs we sent, AND Elastic provide a lot of free training and webinars I would take advantage of those. > **[Free on-demand Elasticsearch and Kibana Training](https://www.elastic.co/training/free)** > > Elastic offers free introductory training for the Elastic (ELK) Stack - Elasticsearch, Kibana, Beats and Logstash. Learn the fundamentals of observability (logging, metrics, APM), security, SIEM, machine learning, & more with on-demand training Good Luck! ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 28, 2021, 2:27pm UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/15 "2021-06-28T14:27:47Z")
Yes I read them all. But actually nothing dropped the pressure sadly. ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/3/6/3663932fa2e95922dcb5266a6ae1f80917f69bba.png) So the shards are now good. Just 86 shards anymore. But pressure is still at a high level: ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/d/6/d69b133f9915e03325a13f5add7dd71e00619a35.png) So the only way now is upgrading the server or running it on-prem, correct? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 28, 2021, 2:38pm UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/16 "2021-06-28T14:38:10Z")
Its going to stay a bit high with such small nodes... quite literally there is only 512MB jvm on each node.. that is they smallest you can possibly run. Remember, the goal is about 10-20 Shards per 1 GB JVM : you have 1.5GB JVM so the goal would be about 30 Shards to you are still 2X. Again these are not hard rules, and with such a limited amount of JVM RAM it going to be tight. And in general your indices are still extremely small, each indices / shard takes memory space in the JVM. I would suggest scaling the cluster up to 2GB or 4 GB nodes. You can do that from the deployment screen just hit edit change the setting and apply, it will take a few minutes it will do a rolling change with no down time. Me... Smallest I ever run is 4GB nodes, but that is just me, you can try 2 GB nodes first. You can scale these cluster up to HUGE Terabytes of RAM and 100s of TBs of storages, which you clearly do not need at this point. No reason to think about on-prem at this point. ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 29, 2021, 8:05am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/17 "2021-06-29T08:05:48Z")
That sounds good. So now I try to reindex. But getting the known error. ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/6/5/651e5b7c6185551efc56a3e8628880294f888a65.png) I now enable the autoscale to get the reindex working properly. Hopefully that fixes it. ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [June 29, 2021, 8:11am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/18 "2021-06-29T08:11:37Z")
Sadly not. So how to reindex when always getting this `circuit` error? ![grafik](https://us1.discourse-cdn.com/elastic/original/3X/c/f/cf167666534f99e8de4b2e46e79fbdc859104c18.png) So autoscale is not working. Should I manually for the reindex upgrade from 1 GB RAM to 4 GB RAM? ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [June 29, 2021, 1:43pm UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/19 "2021-06-29T13:43:27Z")
Autoscale is based on disk usage today so that is not the correct way to fix this issue. (in the near future it will look at memory and CPU pressure. You need to manually scale your cluster. Go into the Elastic Cloud Console and click on your deployment. Click Edit Make the changes And Save at the Bottom. It will take a few minutes Then Try your reindex again. ![Screen Shot 2021-06-29 at 6.41.35 AM](https://us1.discourse-cdn.com/elastic/original/3X/6/a/6a536e7bc8a0fc257d55677a4474223b5d8d26f9.png) ---
**Author:** ![kwoxer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwoxer/32/74809_2.png) [@kwoxer](https://discuss.elastic.co/u/kwoxer)\ **Post date:** [July 1, 2021, 8:03am UTC](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896/20 "2021-07-01T08:03:06Z")
Ok upgrading to 4 GB RAM worked perfectly. Now I just reindex everything without trouble. Afterwards gone back to 1 GB RAM. Then Kibana totally crashed. Already tried to go back to a snapshot. No change. Kibana internal error all the time. ![2021-07-01_09_40_23-greenshot](https://us1.discourse-cdn.com/elastic/original/3X/9/e/9eb5bf551920c6cd2a7967e37d7b70f554b1e6d9.png) [Next page](https://discuss.elastic.co/t/what-is-the-best-mix-of-index-and-shards-on-elastic-cloud/276896.md?page=2)