# How to achieve good performance (huge daily indices) **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2020, 4:02pm UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270 "2020-01-08T16:02:04Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Naseem](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@Naseem](https://discuss.elastic.co/u/Naseem)\ **Post date:** [January 8, 2020, 4:02pm UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270/1 "2020-01-08T16:02:04Z")
Hello, Our free text search is unusable. We have a 3 master + 4 data/ingester setup. We have 1 primary shard and 1 replica per index. Indices are appended by date, and a new one is created every day (typical). The daily indices are huge (\>400GB). Furthermore, the data plane resources are not being used efficiently. 2 of the data nodes are maxing out their requested (kubernetes) 3 CPU cores and utilizing much more disk space than 2 other data nodes that sit idle and are using less disk space. What are we doing wrong? Should we increase replicas, primary shard count, both? Change our rollover strategy? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 8, 2020, 4:12pm UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270/2 "2020-01-08T16:12:40Z")
400gb for one single shard is probably too much. May I suggest you look at the following resources about sizing: [https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing) > **[How many shards should I have in my Elasticsearch cluster?](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)** > > If you are looking for practical guidelines around how many indices and shards to have in your cluster, this blog post will help you avoid common pitfalls. https://www.slideshare.net/slideshow/embed_code/key/vR0XKDq4TGa77z And [https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right](https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right) ---
**Author:** ![Naseem](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@Naseem](https://discuss.elastic.co/u/Naseem)\ **Post date:** [January 9, 2020, 3:49pm UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270/3 "2020-01-09T15:49:08Z")
Thanks @dadoonet, So should we reduce our index size? These are daily indices which seem to be recommended as per your slide deck. Does 1 primary + 1 replica seem right? Or should we increase primaries or replicas? I understand your slide deck, more primaries increase write performance but more replicas increase read performance, is that right? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 10, 2020, 12:03am UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270/4 "2020-01-10T00:03:04Z")
Correct. I'd probably try with 400/50 primaries. So 8 primaries. ---
**Author:** ![Naseem](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@Naseem](https://discuss.elastic.co/u/Naseem)\ **Post date:** [January 10, 2020, 1:12am UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270/5 "2020-01-10T01:12:26Z")
Would rolling over at 50GB per index be an alternative? 1 primary + 1 replica with each index being no more than 50GB? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [January 10, 2020, 6:04am UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270/6 "2020-01-10T06:04:22Z")
We normally advice to keep shard size between 20 to 50 gb. But it depends on your use case. So you need to test it. ---
**Author:** ![Naseem](https://avatars.discourse-cdn.com/v4/letter/n/9fc348/32.png) [@Naseem](https://discuss.elastic.co/u/Naseem)\ **Post date:** [January 10, 2020, 6:14am UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270/7 "2020-01-10T06:14:37Z")
Alright, we will try 8 primaries with 1 replica each, thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 7, 2020, 6:14am UTC](https://discuss.elastic.co/t/how-to-achieve-good-performance-huge-daily-indices/214270/8 "2020-02-07T06:14:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.