# ElasticSearch Performance **URL:** \ **Category:** Elasticsearch\ **Created:** [September 9, 2020, 2:12pm UTC](https://discuss.elastic.co/t/elasticsearch-performance/248026 "2020-09-09T14:12:58Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![oliverlucas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oliverlucas/32/75353_2.png) [@oliverlucas](https://discuss.elastic.co/u/oliverlucas)\ **Post date:** [September 9, 2020, 2:12pm UTC](https://discuss.elastic.co/t/elasticsearch-performance/248026/1 "2020-09-09T14:12:59Z")
Hi All, So quite new to ElasticSearch, and very new to large amounts of data that we seem to be collecting, and because of this i have a few problems that hopefully someone can help with. Background: - We're collecting about ~5 million documents a day, on a 4 node cluster, 2 "hot" and 2 "warm" (Yes, it's under resourced but I don't want to pay the additional costs for more nodes!) - We are using ElasticCloud for this within AWS, I believe the 2 hot nodes have 8GB RAM and the 2 warm nodes have 4GB each. - The storage allocated to the cluster is about 1.5TB or something yet, that doesn't appear to be too usable So what seems to work fine is the writing data to the cluster, I've never seen it in any other state than green which is good, and the memory / CPU of the boxes does not appear to be stressed however searches and functions take ages, literally ages to run, some just never complete and I get an error. One of the advisories is to limit the number of shards on the cluster, each index has 2 shards and I have daily indices but only looking to keep 90 days worth at a max! First question of many.. Would it be better (for search) to have weekly indices even though they would be huge? Thanks Oliver ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 9, 2020, 2:26pm UTC](https://discuss.elastic.co/t/elasticsearch-performance/248026/2 "2020-09-09T14:26:18Z")
It depends what "huge" means. In general try to have a shard size between 20 to 50gb. ---
**Author:** ![oliverlucas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oliverlucas/32/75353_2.png) [@oliverlucas](https://discuss.elastic.co/u/oliverlucas)\ **Post date:** [September 10, 2020, 10:07am UTC](https://discuss.elastic.co/t/elasticsearch-performance/248026/3 "2020-09-10T10:07:07Z")
Hi David, At the moment we're getting ~9GB and ~13GB a day, I guess if we combined them to weekly they could be anywhere from 100GB upwards.. so .. I guess that is what i'd call huge! Are there any "tweaks" you could recommend to get things searching better, and not constantly throwing up errors? Thanks Oliver ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 14, 2020, 10:31am UTC](https://discuss.elastic.co/t/elasticsearch-performance/248026/4 "2020-09-14T10:31:35Z")
I'd recommend looking at [https://www.elastic.co/guide/en/elasticsearch/reference/7.9/ilm-rollover.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/ilm-rollover.html) and define `max_size` to let say something like `30gb`... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 12, 2020, 10:31am UTC](https://discuss.elastic.co/t/elasticsearch-performance/248026/5 "2020-10-12T10:31:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.