Where is all my memory? Or how to estimate better

Hello,

So I've replaced some machines we had with more memory and it seems to have
helped but I'm starting to see some high memory usage on one of the
machines. What I don't understand and can't figure out is where that
memory is being used?

We have 26GB RAM, allocating 13.5G for the ES_HEAP_SIZE. Looking at top
it currently shows the java process to be using (RES) 22G and using free
I confirm that there is only 3172M available.

One would conclude that ES is using it all as shown by top. And yes that
makes sense, but it's not translating to performance of my queries by
caching them and even more so, when I look in marvel I see the following
stats for that one node.

JVM Heap Usage: 36%
OS Memory: 87%

And here's the interesting parts:

Indices Field Data: 427,577,836
Indices Filter Cache: 54,641,296
Indices Lucene Memory: 308,679,803

Field Data Circuit Breaker Estimated Size: 428,615,926
No field data evictions at all. 0.00

So where then is all my memory being used? This doesn't add up to 22GB -
not even close.

Thanks for all your help!
Daniel

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1300cc7b-016f-4a3b-8df9-7b041e80607a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

What OS / version is this?

Jörg

On Fri, Jul 25, 2014 at 4:10 PM, Daniel Schonfeld downwindabeam@gmail.com
wrote:

Hello,

So I've replaced some machines we had with more memory and it seems to
have helped but I'm starting to see some high memory usage on one of the
machines. What I don't understand and can't figure out is where that
memory is being used?

We have 26GB RAM, allocating 13.5G for the ES_HEAP_SIZE. Looking at top
it currently shows the java process to be using (RES) 22G and using free
I confirm that there is only 3172M available.

One would conclude that ES is using it all as shown by top. And yes that
makes sense, but it's not translating to performance of my queries by
caching them and even more so, when I look in marvel I see the following
stats for that one node.

JVM Heap Usage: 36%
OS Memory: 87%

And here's the interesting parts:

Indices Field Data: 427,577,836
Indices Filter Cache: 54,641,296
Indices Lucene Memory: 308,679,803

Field Data Circuit Breaker Estimated Size: 428,615,926
No field data evictions at all. 0.00

So where then is all my memory being used? This doesn't add up to 22GB -
not even close.

Thanks for all your help!
Daniel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1300cc7b-016f-4a3b-8df9-7b041e80607a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1300cc7b-016f-4a3b-8df9-7b041e80607a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGAuY2GDKJuUo9HQ_iJ4bjLOYHZiUeWbkhQ13N81dhXSw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

CoreOS, 367.1.0. Docker 1.0.1 and ES 1.2.2

I don't know if it matters but we are using this plugin which is not
written by
elasticsearch: GitHub - Asquera/elasticsearch-http-basic: HTTP Basic Authentication for Elasticsearch (Discontinued)

On Friday, July 25, 2014 11:02:40 AM UTC-4, Jörg Prante wrote:

What OS / version is this?

Jörg

On Fri, Jul 25, 2014 at 4:10 PM, Daniel Schonfeld <downwi...@gmail.com
<javascript:>> wrote:

Hello,

So I've replaced some machines we had with more memory and it seems to
have helped but I'm starting to see some high memory usage on one of the
machines. What I don't understand and can't figure out is where that
memory is being used?

We have 26GB RAM, allocating 13.5G for the ES_HEAP_SIZE. Looking at
top it currently shows the java process to be using (RES) 22G and using
free I confirm that there is only 3172M available.

One would conclude that ES is using it all as shown by top. And yes that
makes sense, but it's not translating to performance of my queries by
caching them and even more so, when I look in marvel I see the following
stats for that one node.

JVM Heap Usage: 36%
OS Memory: 87%

And here's the interesting parts:

Indices Field Data: 427,577,836
Indices Filter Cache: 54,641,296
Indices Lucene Memory: 308,679,803

Field Data Circuit Breaker Estimated Size: 428,615,926
No field data evictions at all. 0.00

So where then is all my memory being used? This doesn't add up to 22GB -
not even close.

Thanks for all your help!
Daniel

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1300cc7b-016f-4a3b-8df9-7b041e80607a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1300cc7b-016f-4a3b-8df9-7b041e80607a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/000000af-cb3d-4075-a410-955bc8a96c74%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I should also mention that machine which is now low on memory and how
gotten worse, is suffering form high CPU usage too which comes right on
time with the memory getting low.

It's a 4 core machine. Google compute engine - n1-highmem-4.

On Friday, July 25, 2014 1:19:24 PM UTC-4, Daniel Schonfeld wrote:

CoreOS, 367.1.0. Docker 1.0.1 and ES 1.2.2

I don't know if it matters but we are using this plugin which is not
written by elasticsearch:
GitHub - Asquera/elasticsearch-http-basic: HTTP Basic Authentication for Elasticsearch (Discontinued)

On Friday, July 25, 2014 11:02:40 AM UTC-4, Jörg Prante wrote:

What OS / version is this?

Jörg

On Fri, Jul 25, 2014 at 4:10 PM, Daniel Schonfeld downwi...@gmail.com
wrote:

Hello,

So I've replaced some machines we had with more memory and it seems to
have helped but I'm starting to see some high memory usage on one of the
machines. What I don't understand and can't figure out is where that
memory is being used?

We have 26GB RAM, allocating 13.5G for the ES_HEAP_SIZE. Looking at
top it currently shows the java process to be using (RES) 22G and using
free I confirm that there is only 3172M available.

One would conclude that ES is using it all as shown by top. And yes
that makes sense, but it's not translating to performance of my queries by
caching them and even more so, when I look in marvel I see the following
stats for that one node.

JVM Heap Usage: 36%
OS Memory: 87%

And here's the interesting parts:

Indices Field Data: 427,577,836
Indices Filter Cache: 54,641,296
Indices Lucene Memory: 308,679,803

Field Data Circuit Breaker Estimated Size: 428,615,926
No field data evictions at all. 0.00

So where then is all my memory being used? This doesn't add up to 22GB -
not even close.

Thanks for all your help!
Daniel

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1300cc7b-016f-4a3b-8df9-7b041e80607a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1300cc7b-016f-4a3b-8df9-7b041e80607a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f7b80553-93d8-403b-803a-ef48b1a0c92f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.