# Is it required to set MetaSpace Limit for ElasticSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [October 19, 2018, 6:02am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109 "2018-10-19T06:02:05Z")\ **Posts on this page:** 13\ **Page:** 2
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 19, 2018, 7:15am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/22 "2018-10-19T07:15:49Z")
When it comes to how much data you can hold on a node it often comes down to how well you manage heap usage, as this is likely to be the limiting factor. It can therefore vary from use-case to use case. With a 15GB heap I think you are unlikely to get to 10TB, which means that you either end up with smaller or fewer shards. ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [October 19, 2018, 7:22am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/23 "2018-10-19T07:22:19Z")
_A good rule-of-thumb is to ensure you keep the number of shards per node below 20 to 25 per GB heap it has configured_ So 15GB Heap size multiplied by minimum of 20 shards means I can have minimum 300 shards in a node with 30GB RAM _Aim to keep the average shard size between a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size_ If I use the minimum 20GB per shard multiplied by minimum number of 300 shards gives be 6TB of minimum. > And I have a replication factor of 2 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 19, 2018, 7:24am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/24 "2018-10-19T07:24:31Z")
It is not a minimum, it is a range for a maximum. The most common problem we see is that users end up having far too many small shards, which is why we have provided a recommendation around the maximum number of shards you should have. If you are using large shards you are likely to be below this recommended maximum number of shards per node. ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [October 19, 2018, 7:27am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/25 "2018-10-19T07:27:47Z")
_A good rule-of-thumb is to ensure you keep the number of shards per node below 20 to 25 per GB heap it has configured_ So 25 is maximum. That means 375 shards _Aim to keep the average shard size between a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size_ 40GB is maximum. So 375\*40 = 15000 = 14.6TB Disk space is maximum ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [October 19, 2018, 7:28am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/26 "2018-10-19T07:28:01Z")
My shard size is 35GB currently ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 19, 2018, 7:30am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/27 "2018-10-19T07:30:00Z")
How can you have 11622 shards of 35GB (397TB) on 15 nodes with 2TB of disk each? ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [October 19, 2018, 7:31am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/28 "2018-10-19T07:31:23Z")
There are many shards that are very small(single shard single index) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 19, 2018, 7:34am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/29 "2018-10-19T07:34:36Z")
According to the statistics, your average shard size is around 1.6GB. If you want to store data efficiently I would say this is on the small side. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 19, 2018, 7:43am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/31 "2018-10-19T07:43:19Z")
The large number of small shards you have, which brings down the average size, will consume resources and affect cluster performance. I would recommend reducing and/or consolidating these, e.g. by changing the number of primary shards, consolidating indices and/or switching to weekly or monthly indices instead of daily. ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [October 19, 2018, 7:44am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/32 "2018-10-19T07:44:31Z")
Thank you so much @Christian_Dahlqvist. Your help is much appreciated 🙂 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 19, 2018, 7:47am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/33 "2018-10-19T07:47:24Z")
If you are interested in getting the most from your nodes with respect to storage, we are planning a [webinar in November](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch) that may be of interest. ---
**Author:** ![jibsonline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jibsonline/32/63154_2.png) [@jibsonline](https://discuss.elastic.co/u/jibsonline)\ **Post date:** [October 19, 2018, 7:48am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/34 "2018-10-19T07:48:14Z")
Sure.... Thank you so much ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 16, 2018, 7:53am UTC](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109/35 "2018-11-16T07:53:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/is-it-required-to-set-metaspace-limit-for-elasticsearch/153109.md?page=1)