# Anecdotal Advice on memory budget / OOM recovery mechanism for approx 2 billion log lines?

**URL:** <https://discuss.elastic.co/t/anecdotal-advice-on-memory-budget-oom-recovery-mechanism-for-approx-2-billion-log-lines/13162>\
**Category:** Elasticsearch\
**Created:** [August 10, 2013, 7:08pm UTC](https://discuss.elastic.co/t/anecdotal-advice-on-memory-budget-oom-recovery-mechanism-for-approx-2-billion-log-lines/13162 "2013-08-10T19:08:41Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Joseph\_Smith](https://avatars.discourse-cdn.com/v4/letter/j/aeb1de/32.png) [@Joseph\_Smith](https://discuss.elastic.co/u/Joseph_Smith)\
**Post date:** [August 10, 2013, 7:08pm UTC](https://discuss.elastic.co/t/anecdotal-advice-on-memory-budget-oom-recovery-mechanism-for-approx-2-billion-log-lines/13162/1 "2013-08-10T19:08:41Z")

</div>

This is my setup:

-Running ES 0.90.2  
-5 ES nodes, each with 24GB of RAM, 16GB to ES  
-soft field cache  
-7 indices, one for each day of the week, totaling around 2 billion log  
lines  
-doing some amount of faceting / sorting because I am using Kibana for a  
front end  
-log lines are generally small (\< a few K), occasionally there will be some  
larger entries (maybe 100K)

I am getting java heap out of memory exceptions as I approach 2 billion  
lines.

I have three questions:

1. It's my understanding that by setting the field cache to 'soft' the  
field cache entries prevents the field cache itself from causing out of  
memory issue because the GC always has the option to collect the soft  
references. Is this correct?

2. Does anyone have anecdotal advice for the budgets they use, assuming  
they use a similar setup?

3. Obviously avoiding OOM is the ideal, but in case that it does happens  
what's the easiest way to automatically recover? The best I have is polling  
the status API and if it timesout or goes red, then restarting all of the  
ES instances.

Thanks in advance,  
~joe!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:21am UTC](https://discuss.elastic.co/t/anecdotal-advice-on-memory-budget-oom-recovery-mechanism-for-approx-2-billion-log-lines/13162/2 "2017-07-06T02:21:44Z")

</div>

Closed on July 6, 2017, 2:21am UTC