# Understanding the effects of "low memory" (not OOM) on nodes (or: should I just add a new node to my cluster and get on with my life?!)

**URL:** <https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565>\
**Category:** Elasticsearch\
**Created:** [February 1, 2012, 3:20pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565 "2012-02-01T15:20:34Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\
**Post date:** [February 1, 2012, 3:20pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/1 "2012-02-01T15:20:34Z")

</div>

Summary: After loading facets, CPU runs high at idle even with a fair  
bit of memory left - is this expected? And can someone sanity check my  
field cache sizes?

My pseudo-operational cluster consists of 3 nodes, each with 10GB of  
memory allocated (and mlockall set - in any case I have no swap space)

When my entire document set is loaded (2.5M top-level documents,  
various embedded and nested sub-objects), bigdesk shows the store  
sizes as: 6.1GB, 4.3GB and 3.7GB (these sizes vary from re-index to re-  
index as you'd expect, but that sort of distribution is typical). Each  
shard has 1 replica (60 shards in total).

The fun starts when I start to run facets.

First off, I perform a term facet over all geo elements stored in the  
documents (the "total" field = 2163442). In terms of field cache, the  
3 nodes then have the following usage: 2.8GB, 2.5GB, 1GB

First question: does that seem about right for the field cache taken  
up by 2M 14-character strings? That comes out as ~3KB per geo field  
instance, much more per unique token value.

Next I perform a term facet on a different string field (average  
length ~64B), part of a "nested" object ("total"=1809227). This  
increases the field cache usage to: 6.3GB, 3.3GB and 2.8GB.

Looking at "top" on the three nodes, they have the following memory  
usage (Virt/Res in GB): 10.1/8.1, 10.1/6.2, 10.1/5.8.

The node with 10.1/8.1 now uses 40% CPU at idle. Nothing is logged  
(debug level), and none of the timed events in nodes stats show  
anything unusual. Queries still return normally, though performance is  
noticeably degraded (the average search time jumps by 1s or so).

I tried running "index.cache.field.type" as both "resident" (ie  
default) and "soft". The behavior seems the same in both cases.

For what it's worth, the "offending" node contains all 10 shards from  
the largest index, though only 3 of them are "primary".

Second question: Is this expected behavior?

Bonus questions: If so, does it just indicate it's time to add a new  
node or do I have any other options left? Is there any other way of  
detecting this state other than monitoring CPU trends (ie to send an  
alert email out)?

Many thanks for any insight anyone can provide!

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 5, 2012, 10:40am UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/2 "2012-02-05T10:40:26Z")

</div>

Hey,

Trying to analyze high CPU is a tricky business… . I don't see a reason why when no requests are issued you will see this behavior (specifically with the (default) resident type). If you do simulate it again, can you use jstack ([jstack - Stack Trace](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)) and gist the stack trace that you get? It might help narrowing down why it happens.

On Wednesday, February 1, 2012 at 5:20 PM, Alex at Ikanow wrote:

> Summary: After loading facets, CPU runs high at idle even with a fair  
> bit of memory left - is this expected? And can someone sanity check my  
> field cache sizes?
> 
> My pseudo-operational cluster consists of 3 nodes, each with 10GB of  
> memory allocated (and mlockall set - in any case I have no swap space)
> 
> When my entire document set is loaded (2.5M top-level documents,  
> various embedded and nested sub-objects), bigdesk shows the store  
> sizes as: 6.1GB, 4.3GB and 3.7GB (these sizes vary from re-index to re-  
> index as you'd expect, but that sort of distribution is typical). Each  
> shard has 1 replica (60 shards in total).
> 
> The fun starts when I start to run facets.
> 
> First off, I perform a term facet over all geo elements stored in the  
> documents (the "total" field = 2163442). In terms of field cache, the  
> 3 nodes then have the following usage: 2.8GB, 2.5GB, 1GB
> 
> First question: does that seem about right for the field cache taken  
> up by 2M 14-character strings? That comes out as ~3KB per geo field  
> instance, much more per unique token value.
> 
> Next I perform a term facet on a different string field (average  
> length ~64B), part of a "nested" object ("total"=1809227). This  
> increases the field cache usage to: 6.3GB, 3.3GB and 2.8GB.
> 
> Looking at "top" on the three nodes, they have the following memory  
> usage (Virt/Res in GB): 10.1/8.1, 10.1/6.2, 10.1/5.8.
> 
> The node with 10.1/8.1 now uses 40% CPU at idle. Nothing is logged  
> (debug level), and none of the timed events in nodes stats show  
> anything unusual. Queries still return normally, though performance is  
> noticeably degraded (the average search time jumps by 1s or so).
> 
> I tried running "index.cache.field.type" as both "resident" (ie  
> default) and "soft". The behavior seems the same in both cases.
> 
> For what it's worth, the "offending" node contains all 10 shards from  
> the largest index, though only 3 of them are "primary".
> 
> Second question: Is this expected behavior?
> 
> Bonus questions: If so, does it just indicate it's time to add a new  
> node or do I have any other options left? Is there any other way of  
> detecting this state other than monitoring CPU trends (ie to send an  
> alert email out)?
> 
> Many thanks for any insight anyone can provide!

---

<div class="post-metadata">

**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\
**Post date:** [February 6, 2012, 11:41pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/3 "2012-02-06T23:41:31Z")

</div>

Thanks - it actually stopped after a few days in "soft mode", the RES  
memory on all three nodes is \>85% of the VIRT but the field cache  
sizes have reduced to 3.4GB, 1.8GB, 0.7GB with lots of cache  
evictions, interesting (search cache booted out field cache cos nobody  
was doing facets on the big indexes perhaps?) ... when nobody's using  
it and I get the chance, I'll restart and repeat my steps to make it  
happen again.

By the way - for anyone interested in estimating the field cache  
memory usage, I had a look at the code and it appears to be (I assume  
per shard - not per index - and per replica, though I didn't check) ~

8B \* num\_docs \* max(distinct\_field\_values\_per\_doc)

Shay - can you confirm?! ([https://github.com/elasticsearch/](https://github.com/elasticsearch/)  
elasticsearch/blob/master/src/main/java/org/elasticsearch/index/field/  
data/support/FieldDataLoader.java)

(eg if you had 1 doc with 1 geotag, and 1 doc with 1000 geotags, in an  
index with 1 shard and no replicas, the memory usage would be  
8\*(1+1)\*1000=16K)

In my 2 cases above (just looking at my largest index, which has 1  
replica (==2 instances), containing 2.3M/2.5M documents):

- geo (1 doc has 154 geotags) = 8B \* 2.3M \* 154 \* 2 = 5.6GB (I  
actually seemed to use 6.3GB)
- nested field (1 doc has 335 values) = 8B \* 2.3M \* 335 \* 2 = 12.3GB  
(I actually seemed to use only 6.1GB extra no top of geo, not sure if  
nested facets work differently though)

So it doesn't quite match up ... but assuming memory usage is actually  
per shard you should usually do better in practice, depending on the  
distribution of array length to documents.

If this is remotely accurate, the moral of the story would be that: if  
you expect documents to have mostly a small number of values for a  
given field, but with the occasional peak (aka most distributions!),  
you could massively save on memory by allocating the peaks into their  
own index. (or more easily but suboptimally, just putting them all in  
1 shard using the "?routing" parameter?!)

---

<div class="post-metadata">

**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\
**Post date:** [February 7, 2012, 12:03am UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/4 "2012-02-07T00:03:13Z")

</div>

(Edit: I'd forgotten I'd chopped out a load of uninteresting values  
from my nested field - in fact the second max number was 116 not 335,  
so the estimated memory usages are actually both pretty close to the  
actual values, phew!)

---

<div class="post-metadata">

**Author:** ![prashant5375](https://avatars.discourse-cdn.com/v4/letter/p/4491bb/32.png) [@prashant5375](https://discuss.elastic.co/u/prashant5375)\
**Post date:** [February 7, 2012, 5:39am UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/5 "2012-02-07T05:39:27Z")

</div>

Sorry to interrupt in this thread , but i am surprised why you are  
not replying to my query. I have asked my query 10 times but still no  
answer , but u r busy with other queries , is there any body else who can  
answer my query...  
please let me know. References (Bounding Box query is very slow)  
Regards  
Prashant

On Sun, Feb 5, 2012 at 4:10 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:

> Hey,
> 
> Trying to analyze high CPU is a tricky business… . I don't see a reason  
> why when no requests are issued you will see this behavior (specifically  
> with the (default) resident type). If you do simulate it again, can you use  
> jstack (  
> [jstack - Stack Trace](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)) and  
> gist the stack trace that you get? It might help narrowing down why it  
> happens.
> 
> On Wednesday, February 1, 2012 at 5:20 PM, Alex at Ikanow wrote:
> 
> Summary: After loading facets, CPU runs high at idle even with a fair  
> bit of memory left - is this expected? And can someone sanity check my  
> field cache sizes?
> 
> My pseudo-operational cluster consists of 3 nodes, each with 10GB of  
> memory allocated (and mlockall set - in any case I have no swap space)
> 
> When my entire document set is loaded (2.5M top-level documents,  
> various embedded and nested sub-objects), bigdesk shows the store  
> sizes as: 6.1GB, 4.3GB and 3.7GB (these sizes vary from re-index to re-  
> index as you'd expect, but that sort of distribution is typical). Each  
> shard has 1 replica (60 shards in total).
> 
> The fun starts when I start to run facets.
> 
> First off, I perform a term facet over all geo elements stored in the  
> documents (the "total" field = 2163442). In terms of field cache, the  
> 3 nodes then have the following usage: 2.8GB, 2.5GB, 1GB
> 
> First question: does that seem about right for the field cache taken  
> up by 2M 14-character strings? That comes out as ~3KB per geo field  
> instance, much more per unique token value.
> 
> Next I perform a term facet on a different string field (average  
> length ~64B), part of a "nested" object ("total"=1809227). This  
> increases the field cache usage to: 6.3GB, 3.3GB and 2.8GB.
> 
> Looking at "top" on the three nodes, they have the following memory  
> usage (Virt/Res in GB): 10.1/8.1, 10.1/6.2, 10.1/5.8.
> 
> The node with 10.1/8.1 now uses 40% CPU at idle. Nothing is logged  
> (debug level), and none of the timed events in nodes stats show  
> anything unusual. Queries still return normally, though performance is  
> noticeably degraded (the average search time jumps by 1s or so).
> 
> I tried running "index.cache.field.type" as both "resident" (ie  
> default) and "soft". The behavior seems the same in both cases.
> 
> For what it's worth, the "offending" node contains all 10 shards from  
> the largest index, though only 3 of them are "primary".
> 
> Second question: Is this expected behavior?
> 
> Bonus questions: If so, does it just indicate it's time to add a new  
> node or do I have any other options left? Is there any other way of  
> detecting this state other than monitoring CPU trends (ie to send an  
> alert email out)?
> 
> Many thanks for any insight anyone can provide!

---

<div class="post-metadata">

**Author:** ![prashant5375](https://avatars.discourse-cdn.com/v4/letter/p/4491bb/32.png) [@prashant5375](https://discuss.elastic.co/u/prashant5375)\
**Post date:** [February 7, 2012, 5:41am UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/6 "2012-02-07T05:41:03Z")

</div>

This is for "Shay Banon"

On Feb 7, 10:39 am, PS [prashant.vi...@gmail.com](mailto:prashant.vi...@gmail.com) wrote:

> Sorry to interrupt in this thread , but i am surprised why you are  
> not replying to my query. I have asked my query 10 times but still no  
> answer , but u r busy with other queries , is there any body else who can  
> answer my query...  
> please let me know. References (Bounding Box query is very slow)  
> Regards  
> Prashant
> 
> On Sun, Feb 5, 2012 at 4:10 PM, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > Hey,
> 
> > Trying to analyze high CPU is a tricky business… . I don't see a reason  
> > why when no requests are issued you will see this behavior (specifically  
> > with the (default) resident type). If you do simulate it again, can you use  
> > jstack (  
> > [jstack - Stack Trace](http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html)) and  
> > gist the stack trace that you get? It might help narrowing down why it  
> > happens.
> 
> > On Wednesday, February 1, 2012 at 5:20 PM, Alex at Ikanow wrote:
> 
> > Summary: After loading facets, CPU runs high at idle even with a fair  
> > bit of memory left - is this expected? And can someone sanity check my  
> > field cache sizes?
> 
> > My pseudo-operational cluster consists of 3 nodes, each with 10GB of  
> > memory allocated (and mlockall set - in any case I have no swap space)
> 
> > When my entire document set is loaded (2.5M top-level documents,  
> > various embedded and nested sub-objects), bigdesk shows the store  
> > sizes as: 6.1GB, 4.3GB and 3.7GB (these sizes vary from re-index to re-  
> > index as you'd expect, but that sort of distribution is typical). Each  
> > shard has 1 replica (60 shards in total).
> 
> > The fun starts when I start to run facets.
> 
> > First off, I perform a term facet over all geo elements stored in the  
> > documents (the "total" field = 2163442). In terms of field cache, the  
> > 3 nodes then have the following usage: 2.8GB, 2.5GB, 1GB
> 
> > First question: does that seem about right for the field cache taken  
> > up by 2M 14-character strings? That comes out as ~3KB per geo field  
> > instance, much more per unique token value.
> 
> > Next I perform a term facet on a different string field (average  
> > length ~64B), part of a "nested" object ("total"=1809227). This  
> > increases the field cache usage to: 6.3GB, 3.3GB and 2.8GB.
> 
> > Looking at "top" on the three nodes, they have the following memory  
> > usage (Virt/Res in GB): 10.1/8.1, 10.1/6.2, 10.1/5.8.
> 
> > The node with 10.1/8.1 now uses 40% CPU at idle. Nothing is logged  
> > (debug level), and none of the timed events in nodes stats show  
> > anything unusual. Queries still return normally, though performance is  
> > noticeably degraded (the average search time jumps by 1s or so).
> 
> > I tried running "index.cache.field.type" as both "resident" (ie  
> > default) and "soft". The behavior seems the same in both cases.
> 
> > For what it's worth, the "offending" node contains all 10 shards from  
> > the largest index, though only 3 of them are "primary".
> 
> > Second question: Is this expected behavior?
> 
> > Bonus questions: If so, does it just indicate it's time to add a new  
> > node or do I have any other options left? Is there any other way of  
> > detecting this state other than monitoring CPU trends (ie to send an  
> > alert email out)?
> 
> > Many thanks for any insight anyone can provide!

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 7, 2012, 12:06pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/7 "2012-02-07T12:06:31Z")

</div>

Alex, I am not sure I followed your reasoning. The actual computation of the size the field data uses depends on the type of the field data itself (you will see for each type there is a different compute size method).

I did not follow your reasoning on shards and sizing? You mean multi valued fields in the same doc?

On Tuesday, February 7, 2012 at 2:03 AM, Alex at Ikanow wrote:

> (Edit: I'd forgotten I'd chopped out a load of uninteresting values  
> from my nested field - in fact the second max number was 116 not 335,  
> so the estimated memory usages are actually both pretty close to the  
> actual values, phew!)

---

<div class="post-metadata">

**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\
**Post date:** [February 7, 2012, 2:40pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/8 "2012-02-07T14:40:32Z")

</div>

Shay -

I was just looking at multi-valued fields (which will usually dominate  
field cache memory compared to single-valued fields, I think - at  
least that's the case for me by 1-2 orders of magnitude)

From my quick hunt through the code it looked like the dominating  
factor for memory usage for multi-valued field data wasn't the types  
themselves but the "ordinals" array of array of ints created in the  
FieldDataLoader, which mapped the term indices to document ids (and is  
independent to data type).

I assume that in addition to this ordinals member, there is type-  
specific storage, but with memory usage that does not explicitly scale  
with the number of documents, hence irrelevant for larger indexes  
(with multi-valued fields, also I assume for very large indexes with a  
smallish number of distinct terms).

(I also assumed/guess but didn't explicitly check that the field cache  
simply caches these composite objects consisting of type-specific  
storage and the ordinals array using the google cache builder class.)

It seems like my case of having several multi-valued fields, which can  
have large-ish arrays per document is unusual?

Incidentally, \<1% of my documents have a arrays of \>= 25% of the max  
array length (per field), so I can easily get at least 4x memory  
saving by putting the "big" docs in their own index (or shard?)

On Feb 7, 7:06 am, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> Alex, I am not sure I followed your reasoning. The actual computation of the size the field data uses depends on the type of the field data itself (you will see for each type there is a different compute size method).
> 
> I did not follow your reasoning on shards and sizing? You mean multi valued fields in the same doc?
> 
> On Tuesday, February 7, 2012 at 2:03 AM, Alex at Ikanow wrote:
> 
> > (Edit: I'd forgotten I'd chopped out a load of uninteresting values  
> > from my nested field - in fact the second max number was 116 not 335,  
> > so the estimated memory usages are actually both pretty close to the  
> > actual values, phew!)

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 7, 2012, 6:47pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/9 "2012-02-07T18:47:34Z")

</div>

Yea, sadly with multi valued fields per doc you will get a hit in terms of memory usage because we need to keep track of all the references to the values.

One option that you might want to play with, is to use nested documents for all the different values. It will mean more documents in Lucene (since each nested object is another doc in Lucene), but on the other hand, it means no longer having multiple values per doc. In this case, you will need to start using nested queries / filters and nested execution of facets to get the counts you want. I would give it a go on a simple test env to see if it makes a difference.

On Tuesday, February 7, 2012 at 4:40 PM, Alex at Ikanow wrote:

> Shay -
> 
> I was just looking at multi-valued fields (which will usually dominate  
> field cache memory compared to single-valued fields, I think - at  
> least that's the case for me by 1-2 orders of magnitude)
> 
> From my quick hunt through the code it looked like the dominating  
> factor for memory usage for multi-valued field data wasn't the types  
> themselves but the "ordinals" array of array of ints created in the  
> FieldDataLoader, which mapped the term indices to document ids (and is  
> independent to data type).
> 
> I assume that in addition to this ordinals member, there is type-  
> specific storage, but with memory usage that does not explicitly scale  
> with the number of documents, hence irrelevant for larger indexes  
> (with multi-valued fields, also I assume for very large indexes with a  
> smallish number of distinct terms).
> 
> (I also assumed/guess but didn't explicitly check that the field cache  
> simply caches these composite objects consisting of type-specific  
> storage and the ordinals array using the google cache builder class.)
> 
> It seems like my case of having several multi-valued fields, which can  
> have large-ish arrays per document is unusual?
> 
> Incidentally, \<1% of my documents have a arrays of \>= 25% of the max  
> array length (per field), so I can easily get at least 4x memory  
> saving by putting the "big" docs in their own index (or shard?)
> 
> On Feb 7, 7:06 am, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> 
> > Alex, I am not sure I followed your reasoning. The actual computation of the size the field data uses depends on the type of the field data itself (you will see for each type there is a different compute size method).
> > 
> > I did not follow your reasoning on shards and sizing? You mean multi valued fields in the same doc?
> > 
> > On Tuesday, February 7, 2012 at 2:03 AM, Alex at Ikanow wrote:
> > 
> > > (Edit: I'd forgotten I'd chopped out a load of uninteresting values  
> > > from my nested field - in fact the second max number was 116 not 335,  
> > > so the estimated memory usages are actually both pretty close to the  
> > > actual values, phew!)

---

<div class="post-metadata">

**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\
**Post date:** [February 7, 2012, 7:02pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/10 "2012-02-07T19:02:12Z")

</div>

Hmm interestingly, the nested field memory usage seems to follow the  
same pattern (in the above examples, the second field is nested, and  
with 2.3M docs and a max of 113 values, uses 6.1GB vs ~5.3GB  
"expected", basically identical to the non-nested geo) ... perhaps  
that's just coincidence (annoyingly if so, since there seems to be a  
workable solution to reducing memory usage in the non-nested  
case!)? ... What would you expect the approximate memory usage to be  
for a list of nested objects?

When I get a spare moment I'll experiment with forcing documents with  
large arrays into their own shard/index (\*) and report back on the  
memory savings/performance

(\*) Am I right that you cache field data by _shard_ and not by  
_index_?

On Feb 7, 1:47 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:

> Yea, sadly with multi valued fields per doc you will get a hit in terms of memory usage because we need to keep track of all the references to the values.
> 
> One option that you might want to play with, is to use nested documents for all the different values. It will mean more documents in Lucene (since each nested object is another doc in Lucene), but on the other hand, it means no longer having multiple values per doc. In this case, you will need to start using nested queries / filters and nested execution of facets to get the counts you want. I would give it a go on a simple test env to see if it makes a difference.

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [February 7, 2012, 8:56pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/11 "2012-02-07T20:56:52Z")

</div>

On Tue, 2012-02-07 at 11:09 +0530, PS wrote:

> Sorry to interrupt in this thread , but i am surprised why you are not  
> replying to my query. I have asked my query 10 times but still no  
> answer , but u r busy with other queries , is there any body else who  
> can answer my query...

Interrupting another thread and, in fact, posting the same query several  
times is considered to be spam.

Please don't do it.

None of us is being paid to answer your queries.

Perhaps you need to rethink how you are posting your question to make it  
more interesting. Or perhaps your question has already been answered,  
and you need to reread the answers that have been given. Or perhaps you  
need to read through the documentation on the website. Or perhaps you  
need to put together a simple test case that demonstrates the problem an  
open an issue on github.

Whatever the correct answer is, demanding an answer is sure to guarantee  
you a complete lack of interest from the others on the list.

clint

> please let me know. References (Bounding Box query is very slow)  
> Regards  
> Prashant
> 
> On Sun, Feb 5, 2012 at 4:10 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote:  
> Hey,
> 
> ```
> Trying to analyze high CPU is a tricky business… . I don't
> see a reason why when no requests are issued you will see this
> behavior (specifically with the (default) resident type). If
> you do simulate it again, can you use jstack
> (http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jstack.html) and gist the stack trace that you get? It might help narrowing down why it happens.
>     
> On Wednesday, February 1, 2012 at 5:20 PM, Alex at Ikanow
> wrote:
>     
> > 
> > 
> > Summary: After loading facets, CPU runs high at idle even
> > with a fair
> > bit of memory left - is this expected? And can someone
> > sanity check my
> > field cache sizes?
> > 
> > 
> > My pseudo-operational cluster consists of 3 nodes, each with
> > 10GB of
> > memory allocated (and mlockall set - in any case I have no
> > swap space)
> > 
> > 
> > When my entire document set is loaded (2.5M top-level
> > documents,
> > various embedded and nested sub-objects), bigdesk shows the
> > store
> > sizes as: 6.1GB, 4.3GB and 3.7GB (these sizes vary from
> > re-index to re-
> > index as you'd expect, but that sort of distribution is
> > typical). Each
> > shard has 1 replica (60 shards in total).
> > 
> > 
> > The fun starts when I start to run facets.
> > 
> > 
> > First off, I perform a term facet over all geo elements
> > stored in the
> > documents (the "total" field = 2163442). In terms of field
> > cache, the
> > 3 nodes then have the following usage: 2.8GB, 2.5GB, 1GB
> > 
> > 
> > First question: does that seem about right for the field
> > cache taken
> > up by 2M 14-character strings? That comes out as ~3KB per
> > geo field
> > instance, much more per unique token value.
> > 
> > 
> > Next I perform a term facet on a different string field
> > (average
> > length ~64B), part of a "nested" object ("total"=1809227).
> > This
> > increases the field cache usage to: 6.3GB, 3.3GB and 2.8GB.
> > 
> > 
> > Looking at "top" on the three nodes, they have the following
> > memory
> > usage (Virt/Res in GB): 10.1/8.1, 10.1/6.2, 10.1/5.8.
> > 
> > 
> > The node with 10.1/8.1 now uses 40% CPU at idle. Nothing is
> > logged
> > (debug level), and none of the timed events in nodes stats
> > show
> > anything unusual. Queries still return normally, though
> > performance is
> > noticeably degraded (the average search time jumps by 1s or
> > so).
> > 
> > 
> > I tried running "index.cache.field.type" as both
> > "resident" (ie
> > default) and "soft". The behavior seems the same in both
> > cases.
> > 
> > 
> > For what it's worth, the "offending" node contains all 10
> > shards from
> > the largest index, though only 3 of them are "primary".
> > 
> > 
> > Second question: Is this expected behavior?
> > 
> > 
> > Bonus questions: If so, does it just indicate it's time to
> > add a new
> > node or do I have any other options left? Is there any other
> > way of
> > detecting this state other than monitoring CPU trends (ie to
> > send an
> > alert email out)?
> > 
> > 
> > Many thanks for any insight anyone can provide!
> 
> ```

---

<div class="post-metadata">

**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\
**Post date:** [February 8, 2012, 3:11am UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/12 "2012-02-08T03:11:30Z")

</div>

OK I reran my original tests ... the nested field only requires \<1GB  
of field cache so you were of course right about that.

The reason I had my incorrect memory usage in the original email is  
interesting: if I run a geo facet once (from a clean startup), I get  
~6GB usage, but if I run it again (with or without the nested field)  
then I get ~12GB usage (so I had incorrectly assumed the extra 6GB was  
from the nested field whereas actually it was from the second time  
through the geo).

Similarly if I run the nested field facet twice from startup, I get  
200-400MB the first time through, then 650MB the second time through.

Subsequent runs don't seem to increase the memory usage.

Any quick explanation for what's going on here? Eg does it not  
necessarily load all replicas the first time through? The only issue  
with this is that the geo somehow uses 12GB vs my expected 6GB (with 1  
replica), not sure where I can have dropped an x2 in my memory  
calculation... (unless the ordinals array gets copied somewhere?)

My geo array is a duplication of information already stored in nested  
objects (for performance, oops!), though annoyingly it can be used for  
sorting documents, so I need to see how (/how well) that works with  
nested objects.

If sorting and nested objects don't play nice together, I think the  
plan will be to use the geo within the nested objects for faceting,  
the existing un-nested geo-array for sorting and then use a "soft"  
field cache (since geo-sorting is somewhat infrequently used, at least  
in the larger datasets), together with sticking all documents with  
many geo elements into their own index (or maybe shard). I don't  
suppose there's a hidden way of making certain fields use resident  
cache and others use soft cache?

On Feb 7, 2:02 pm, Alex at Ikanow [apigg...@ikanow.com](mailto:apigg...@ikanow.com) wrote:

> Hmm interestingly, the nested field memory usage seems to follow the  
> same pattern (in the above examples, the second field is nested, and  
> with 2.3M docs and a max of 113 values, uses 6.1GB vs ~5.3GB  
> "expected", basically identical to the non-nested geo) ... perhaps  
> that's just coincidence (annoyingly if so, since there seems to be a  
> workable solution to reducing memory usage in the non-nested  
> case!)? ... What would you expect the approximate memory usage to be  
> for a list of nested objects?
> 
> When I get a spare moment I'll experiment with forcing documents with  
> large arrays into their own shard/index (\*) and report back on the  
> memory savings/performance
> 
> (\*) Am I right that you cache field data by _shard_ and not by  
> _index_?
> 
> On Feb 7, 1:47 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote:
> 
> > Yea, sadly with multi valued fields per doc you will get a hit in terms of memory usage because we need to keep track of all the references to the values.
> 
> > One option that you might want to play with, is to use nested documents for all the different values. It will mean more documents in Lucene (since each nested object is another doc in Lucene), but on the other hand, it means no longer having multiple values per doc. In this case, you will need to start using nested queries / filters and nested execution of facets to get the counts you want. I would give it a go on a simple test env to see if it makes a difference.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 9, 2012, 7:06am UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/13 "2012-02-09T07:06:19Z")

</div>

The reason it increases is because the first execution hits one set of shards, and the second hits their counterparts (the replicas). Also, I highly recommend against using the soft field cache in any case. If you the memory is needed, at any time, let it be there.

On Wednesday, February 8, 2012 at 5:11 AM, Alex at Ikanow wrote:

> OK I reran my original tests ... the nested field only requires \<1GB  
> of field cache so you were of course right about that.
> 
> The reason I had my incorrect memory usage in the original email is  
> interesting: if I run a geo facet once (from a clean startup), I get  
> ~6GB usage, but if I run it again (with or without the nested field)  
> then I get ~12GB usage (so I had incorrectly assumed the extra 6GB was  
> from the nested field whereas actually it was from the second time  
> through the geo).
> 
> Similarly if I run the nested field facet twice from startup, I get  
> 200-400MB the first time through, then 650MB the second time through.
> 
> Subsequent runs don't seem to increase the memory usage.
> 
> Any quick explanation for what's going on here? Eg does it not  
> necessarily load all replicas the first time through? The only issue  
> with this is that the geo somehow uses 12GB vs my expected 6GB (with 1  
> replica), not sure where I can have dropped an x2 in my memory  
> calculation... (unless the ordinals array gets copied somewhere?)
> 
> My geo array is a duplication of information already stored in nested  
> objects (for performance, oops!), though annoyingly it can be used for  
> sorting documents, so I need to see how (/how well) that works with  
> nested objects.
> 
> If sorting and nested objects don't play nice together, I think the  
> plan will be to use the geo within the nested objects for faceting,  
> the existing un-nested geo-array for sorting and then use a "soft"  
> field cache (since geo-sorting is somewhat infrequently used, at least  
> in the larger datasets), together with sticking all documents with  
> many geo elements into their own index (or maybe shard). I don't  
> suppose there's a hidden way of making certain fields use resident  
> cache and others use soft cache?
> 
> On Feb 7, 2:02 pm, Alex at Ikanow \<[apigg...@ikanow.com](mailto:apigg...@ikanow.com) ([http://ikanow.com](http://ikanow.com))\> wrote:
> 
> > Hmm interestingly, the nested field memory usage seems to follow the  
> > same pattern (in the above examples, the second field is nested, and  
> > with 2.3M docs and a max of 113 values, uses 6.1GB vs ~5.3GB  
> > "expected", basically identical to the non-nested geo) ... perhaps  
> > that's just coincidence (annoyingly if so, since there seems to be a  
> > workable solution to reducing memory usage in the non-nested  
> > case!)? ... What would you expect the approximate memory usage to be  
> > for a list of nested objects?
> > 
> > When I get a spare moment I'll experiment with forcing documents with  
> > large arrays into their own shard/index (\*) and report back on the  
> > memory savings/performance
> > 
> > (\*) Am I right that you cache field data by _shard_ and not by  
> > _index_?
> > 
> > On Feb 7, 1:47 pm, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote:
> > 
> > > Yea, sadly with multi valued fields per doc you will get a hit in terms of memory usage because we need to keep track of all the references to the values.
> > 
> > > One option that you might want to play with, is to use nested documents for all the different values. It will mean more documents in Lucene (since each nested object is another doc in Lucene), but on the other hand, it means no longer having multiple values per doc. In this case, you will need to start using nested queries / filters and nested execution of facets to get the counts you want. I would give it a go on a simple test env to see if it makes a difference.

---

<div class="post-metadata">

**Author:** ![Alex\_At\_Ikanow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_at_ikanow/32/676_2.png) [@Alex\_At\_Ikanow](https://discuss.elastic.co/u/Alex_At_Ikanow)\
**Post date:** [February 9, 2012, 4:38pm UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/14 "2012-02-09T16:38:58Z")

</div>

Shay - that makes sense, thanks.

The final guesstimate came out as 4B \* num\_docs \*  
max(distinct\_field\_values\_per\_doc) \* 4 (\<- mystery multiplier!) per  
original/replica

From a sw engineering point of view I agree with you re field cache;  
but with my system engineer hat on I have a system to which users can  
upload data (which in the future they will able to run their own  
aggregations over), and which doesn't auto-provision (yet!), therefore  
the requirement to degrade gracefully (which the soft cache does in my  
experience) is very important. I'm willing to live with sub-optimal  
performance in low memory cases (and again fwiw it's worked very well  
so far for me)

Finally, for information (if anyone's still reading!): I changed my  
application logic to put any documents with \>20 geo points in a  
separate index (\<\<1% of all documents), and that reduced my index size  
across the 3 nodes from ~14.0GB down to ~2.2GB, so that's a 4-line  
change well worth doing.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [February 12, 2012, 11:18am UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/15 "2012-02-12T11:18:23Z")

</div>

The field cache is a tricky one, you need it to do computation, but if it grows too large, without having enough memory to accommodate for it, then you get into a problem. The problem is that you can then issue another request that needs to use that field data, and it will need to be loaded again, so a soft cache will be problematic as well, since it can get to a place where it gets loaded and evicted each time.

On Thursday, February 9, 2012 at 6:38 PM, Alex at Ikanow wrote:

> Shay - that makes sense, thanks.
> 
> The final guesstimate came out as 4B \* num\_docs \*  
> max(distinct\_field\_values\_per\_doc) \* 4 (\<- mystery multiplier!) per  
> original/replica
> 
> From a sw engineering point of view I agree with you re field cache;  
> but with my system engineer hat on I have a system to which users can  
> upload data (which in the future they will able to run their own  
> aggregations over), and which doesn't auto-provision (yet!), therefore  
> the requirement to degrade gracefully (which the soft cache does in my  
> experience) is very important. I'm willing to live with sub-optimal  
> performance in low memory cases (and again fwiw it's worked very well  
> so far for me)
> 
> Finally, for information (if anyone's still reading!): I changed my  
> application logic to put any documents with \>20 geo points in a  
> separate index (\<\<1% of all documents), and that reduced my index size  
> across the 3 nodes from ~14.0GB down to ~2.2GB, so that's a 4-line  
> change well worth doing.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:39am UTC](https://discuss.elastic.co/t/understanding-the-effects-of-low-memory-not-oom-on-nodes-or-should-i-just-add-a-new-node-to-my-cluster-and-get-on-with-my-life/6565/16 "2017-07-06T03:39:50Z")

</div>