High memory usage on dedicated master nodes

Elastic Stack Elasticsearch
1

We have cluster with 22 data nodes and 3 dedicated master nodes running ES
1.1.0. Data nodes have 16 GB of memory (half given to JVM heap) and the
dedicated masters have 4 GB (half for heap). Data nodes have consistent
memory usage (about 50-60%) but we're observing that the master node is
constantly growing and shrinking it's memory use (after GC) every 5
minutes. See the bigdesk graph.

https://lh6.googleusercontent.com/-03s6N1l4oQ8/U8ayMdqO86I/AAAAAAAAABg/BWC91tetsvk/s1600/Screen+Shot+2014-07-16+at+1.04.25+PM.png

I'm curious as to why the master node is using up close to 1.5 GB of
memory. I thought that maintaining cluster state shouldn't take up that
much memory and master nodes should be able to work with much less memory.
Anybody know why?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6a273add-a20b-41ab-9941-e6c9197389c0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Are they master only or are you sending queries through them as well?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 17 July 2014 03:14, David Smith davidksmith2k@gmail.com wrote:

We have cluster with 22 data nodes and 3 dedicated master nodes running ES
1.1.0. Data nodes have 16 GB of memory (half given to JVM heap) and the
dedicated masters have 4 GB (half for heap). Data nodes have consistent
memory usage (about 50-60%) but we're observing that the master node is
constantly growing and shrinking it's memory use (after GC) every 5
minutes. See the bigdesk graph.

https://lh6.googleusercontent.com/-03s6N1l4oQ8/U8ayMdqO86I/AAAAAAAAABg/BWC91tetsvk/s1600/Screen+Shot+2014-07-16+at+1.04.25+PM.png

I'm curious as to why the master node is using up close to 1.5 GB of
memory. I thought that maintaining cluster state shouldn't take up that
much memory and master nodes should be able to work with much less memory.
Anybody know why?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6a273add-a20b-41ab-9941-e6c9197389c0%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6a273add-a20b-41ab-9941-e6c9197389c0%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624Zem%3D3TnHCoi7Gta7rCsK%3DDiuwurjwHRxNQ6HLjho9Z1Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Maybe I'm missing something but if you give java mms em it will use it if only to store garbage. The trough after a garbage collection is usually more indicative of what is actually in use.

This looks like a CMS/parnew setup. Parnew is fast and low blocking, but leaves stuff behind. CMS blocks but is really good at taking out the garbage.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0e7c621e-3e85-4bc3-ac0f-a1971cb097f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

They are dedicated masters and no queries are going through them.

smonasco, that's it I believe. It's ParNew for young gen. I made a mistake in our puppet configs and gave the same amount of memory to both data nodes and master nodes for young generation (Xmn) even though master nodes only have 1/4 of the memory data nodes have. So all 2G of memory in these master nodes are in young generation.

On Jul 17, 2014, at 12:07 AM, smonasco smonasco@gmail.com wrote:

Maybe I'm missing something but if you give java mms em it will use it if only to store garbage. The trough after a garbage collection is usually more indicative of what is actually in use.

This looks like a CMS/parnew setup. Parnew is fast and low blocking, but leaves stuff behind. CMS blocks but is really good at taking out the garbage.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0e7c621e-3e85-4bc3-ac0f-a1971cb097f6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/A5405B23-A1AC-4ABC-B357-81B3977A5E39%40gmail.com.
For more options, visit https://groups.google.com/d/optout.