# ES HEAP // ES infrastructure by 10 billion docs per year

**URL:** <https://discuss.elastic.co/t/es-heap-es-infrastructure-by-10-billion-docs-per-year/50535>\
**Category:** Elasticsearch\
**Created:** [May 20, 2016, 7:25am UTC](https://discuss.elastic.co/t/es-heap-es-infrastructure-by-10-billion-docs-per-year/50535 "2016-05-20T07:25:12Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dom\_Sie](https://avatars.discourse-cdn.com/v4/letter/d/9e8a1a/32.png) [@Dom\_Sie](https://discuss.elastic.co/u/Dom_Sie)\
**Post date:** [May 20, 2016, 7:25am UTC](https://discuss.elastic.co/t/es-heap-es-infrastructure-by-10-billion-docs-per-year/50535/1 "2016-05-20T07:25:12Z")

</div>

Hi, we use ES 1.4.4. Our Cluster consists of 10 DataNodes within 16 GB Ram total and 10 GB HEAP.  
We have 174 indices and 1732 shard (5 primary, 5 replica per index) [we made a new index every month]  
Overall we have 818,876,914 docs (replicas are not taken into account) - but we want to expand the system and then save in the future 10 billion docs per year...  
We have 3 master nodes, 2 client nodes and 3 proxy clients and use loadbalancing.  
Our heap usage increases rapidly and also the retrofitting of Ram || increase the HEAP does not bring long-term improvements more.  
Now therefore I ask for advice how to reduce HEAP utilization.

1.) increases the use of the heap linearly with number of docs in the Node / Index / shard?

2.) I have heard the rule: per 100 million docs needed 5-10 GB memory - is this right?

3.) When we generate in the future 10 billion docs per year:  
Is this practicable//possible with es [and a not too large number of datanodes] ?

4.) What are the components/functions of the ES-HEAP usage in general? i mean: which functions/services of ES generally uses the HEAP

5.) which are measures/actions to keep the HEAP usage generally lower?

6.) Would you selectively disable / deactivate specific caches to lower the HEAP-usage

7.) Would you hold the caches persistent?

8.) Are there any best-practice settings for the garbage collector?

THX for your help 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:50pm UTC](https://discuss.elastic.co/t/es-heap-es-infrastructure-by-10-billion-docs-per-year/50535/2 "2017-07-05T22:50:12Z")

</div>