Is it required to set MetaSpace Limit for ElasticSearch

When it comes to how much data you can hold on a node it often comes down to how well you manage heap usage, as this is likely to be the limiting factor. It can therefore vary from use-case to use case.

With a 15GB heap I think you are unlikely to get to 10TB, which means that you either end up with smaller or fewer shards.

A good rule-of-thumb is to ensure you keep the number of shards per node below 20 to 25 per GB heap it has configured

So 15GB Heap size multiplied by minimum of 20 shards means I can have minimum 300 shards in a node with 30GB RAM

Aim to keep the average shard size between a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size

If I use the minimum 20GB per shard multiplied by minimum number of 300 shards gives be 6TB of minimum.

And I have a replication factor of 2

It is not a minimum, it is a range for a maximum. The most common problem we see is that users end up having far too many small shards, which is why we have provided a recommendation around the maximum number of shards you should have. If you are using large shards you are likely to be below this recommended maximum number of shards per node.

A good rule-of-thumb is to ensure you keep the number of shards per node below 20 to 25 per GB heap it has configured

So 25 is maximum. That means 375 shards

Aim to keep the average shard size between a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size

40GB is maximum. So 375*40 = 15000 = 14.6TB Disk space is maximum

My shard size is 35GB currently

How can you have 11622 shards of 35GB (397TB) on 15 nodes with 2TB of disk each?

There are many shards that are very small(single shard single index)

According to the statistics, your average shard size is around 1.6GB. If you want to store data efficiently I would say this is on the small side.

The large number of small shards you have, which brings down the average size, will consume resources and affect cluster performance. I would recommend reducing and/or consolidating these, e.g. by changing the number of primary shards, consolidating indices and/or switching to weekly or monthly indices instead of daily.

1 Like

Thank you so much @Christian_Dahlqvist. Your help is much appreciated :slight_smile:

If you are interested in getting the most from your nodes with respect to storage, we are planning a webinar in November that may be of interest.

1 Like

Sure.... Thank you so much

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.