# Minimum hardware requirement for 500 million documents

**URL:** <https://discuss.elastic.co/t/minimum-hardware-requirement-for-500-million-documents/27924>\
**Category:** Elasticsearch\
**Created:** [August 24, 2015, 7:16am UTC](https://discuss.elastic.co/t/minimum-hardware-requirement-for-500-million-documents/27924 "2015-08-24T07:16:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![puneetm](https://avatars.discourse-cdn.com/v4/letter/p/e480ec/32.png) [@puneetm](https://discuss.elastic.co/u/puneetm)\
**Post date:** [August 24, 2015, 7:16am UTC](https://discuss.elastic.co/t/minimum-hardware-requirement-for-500-million-documents/27924/1 "2015-08-24T07:16:15Z")

</div>

Hi,

I have a document with about 15 fields and I want to index 500million documents (around 350 GB of data) with elastisearch. I also will be querying this data with medium to complex queries including aggregations etc.

Can somebody please suggest the minimum hardware configuration that I should take into account for this ES server with one node?

Thanks  
Puneet

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 24, 2015, 7:43am UTC](https://discuss.elastic.co/t/minimum-hardware-requirement-for-500-million-documents/27924/2 "2015-08-24T07:43:17Z")

</div>

You can easily store that amount of data on a single JVM and the max heap you want for a single JVM is 30GB.  
Ideally you should test to see how your data structure applies to your use case and Elasticsearch.

---

<div class="post-metadata">

**Author:** ![puneetm](https://avatars.discourse-cdn.com/v4/letter/p/e480ec/32.png) [@puneetm](https://discuss.elastic.co/u/puneetm)\
**Post date:** [August 25, 2015, 6:44am UTC](https://discuss.elastic.co/t/minimum-hardware-requirement-for-500-million-documents/27924/3 "2015-08-25T06:44:21Z")

</div>

Hi,

Thanks for the reply.

Following are some of the follow up queries:

I tested with 25 million records, and found that I can query my data with 1GB of allocated heap.  
But, if I increase the records to 30million and then query the data with 1GB of allocated heap, I get the following exception:"

```
    "UncheckedExecutionException[org.elasticsearch.common.breaker.CircuitBreakingException:  
     [FIELDDATA] Data too large, data for [bad_score] would be larger than limit of  
     [623326003/594.4mb]]; nested: CircuitBreakingException[[FIELDDATA] Data too large, data for 
     [----] would be larger than limit of [623326003/594.4mb]]; }
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(TransportSearchTypeAction.java:237)
    at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$1.onFailure(TransportSearchTypeAction.java:183)
    at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:565)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)"

```

1. Please let me know in detail what are the different strategies to avoid this error in the Prod environment, Do we need to keep monitoring the elasticsearch nodes and then scale horizontally as and when required?

2. Can I assume that if I increase my data n times of 25 million then I also need to increase my heap memory by n of 1GB?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [August 25, 2015, 7:19am UTC](https://discuss.elastic.co/t/minimum-hardware-requirement-for-500-million-documents/27924/4 "2015-08-25T07:19:04Z")

</div>

1. Look at [doc values](https://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html).
2. It's more complicated than that, but hard to quantify. You need to test.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:54pm UTC](https://discuss.elastic.co/t/minimum-hardware-requirement-for-500-million-documents/27924/5 "2017-07-05T23:54:04Z")

</div>