# Heap usage vs number of shards
**URL:** https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646
**Category:** Elasticsearch
**Created:** [October 20, 2017, 4:30am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646 "2017-10-20T04:30:41Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@vahissan](https://discuss.elastic.co/u/vahissan)
#### Post date: [October 20, 2017, 4:30am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/1 "2017-10-20T04:30:41Z")
We have a 3 node cluster with 128GB RAM in each node. Heap allocated for each node is 31GB. Our expected monthly record count is 8 billion per month. So we have created daily indices to hold data up to 2 years. Now, after 9 months heap usage always stays around 80% and sometimes all nodes crash at the same time due to OutOfMemory exception. Can the heap usage reduced by reindexing the documents with monthly indices? If so, what kind of heap usage improvement can I expect?
Is there any other way I can reduce heap usage without reducing number of shards?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 20, 2017, 6:10am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/2 "2017-10-20T06:10:14Z")
How many indices and shards do you currently have in the cluster? What is the average shard size in the cluster? Have you read [this blog post around shards and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster)?
---
### Author:  [@vahissan](https://discuss.elastic.co/u/vahissan)
#### Post date: [October 20, 2017, 6:49am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/3 "2017-10-20T06:49:43Z")
Thanks, I will read the blog post. Currently we have 285 indices and 1638 shards. Average shard size is around 8GB.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 20, 2017, 7:39am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/4 "2017-10-20T07:39:04Z")
Have you run [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/indices-forcemerge.html) with `max_num_segments` set to `1` on older indices that are no longer written to?
---
### Author:  [@vahissan](https://discuss.elastic.co/u/vahissan)
#### Post date: [October 20, 2017, 7:44am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/5 "2017-10-20T07:44:31Z")
Thanks. I will check.
---
### Author:  [@vahissan](https://discuss.elastic.co/u/vahissan)
#### Post date: [October 20, 2017, 7:49am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/6 "2017-10-20T07:49:49Z")
There is also a change in requirement to upgrade the cluster to hold 15 year data. In that case, I believe it is important to reindex with monthly indices. Do you agree? I am thinking about 18 nodes with 30GB heap each and 6 primary shards + 1 replica shard. Can I have your advice on this?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 20, 2017, 2:24pm UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/7 "2017-10-20T14:24:44Z")
Are you going to index 15 years worth of data now or keep just keep the data you are indexing now that long?
---
### Author:  [@vahissan](https://discuss.elastic.co/u/vahissan)
#### Post date: [October 20, 2017, 7:31pm UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/8 "2017-10-20T19:31:16Z")
We are going to index 10 years data now, and going to index for 5 more years later on.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 20, 2017, 8:21pm UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/9 "2017-10-20T20:21:46Z")
How large do you estimate a shard for a monthly index would be?
---
### Author:  [@vahissan](https://discuss.elastic.co/u/vahissan)
#### Post date: [October 20, 2017, 8:43pm UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/10 "2017-10-20T20:43:29Z")
Around 300GB for a single shard. I'm planning to have 12 shards per month (6 primary + 1 replica).
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 21, 2017, 7:55am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/11 "2017-10-21T07:55:12Z")
I would recommend performing a benchmark to determine the max shard size [as described in this Elastic{ON} talk](https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing). 300GB is quite large, and may result in slow queries and issues when recovering.
If I calculate correctly, you estimate you will generate about 3.6TB of indexed data per month (primaries and replicas). Over 15 years that is 648TB. To handle that amount of data I suspect you will need considerably more than 18 data nodes.
---
### Author:  [@vahissan](https://discuss.elastic.co/u/vahissan)
#### Post date: [October 22, 2017, 7:52am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/12 "2017-10-22T07:52:11Z")
I still have to watch the talk as I am currently traveling. Just a quick question before I watch - do you think it is better to have nodes with lesser RAM than 128GB (i.e. 64GB) when planning the cluster to hold 15 years of data?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [October 22, 2017, 8:16am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/13 "2017-10-22T08:16:07Z")
When holding lots of data you often want to maximize heap. You could do that by having smaller hosts or simply running 2 Elasticsearch instances on each host. I would recommend spinning up a cluster with a few nodes and run a benchmark to determine exactly how much data you will be able to hold per node based on your expected indexing and query load as described in the video I liked to. This will allow you to estimate how many nodes you will need for that amount of data.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 19, 2017, 8:16am UTC](https://discuss.elastic.co/t/heap-usage-vs-number-of-shards/104646/14 "2017-11-19T08:16:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.