Tuning elasticsearch node client non-heap memory consumption

Elastic Stack Elasticsearch
1

Hello,

We are running a node client on each machine with small JVM heap -Xms384m
-Xmx384m -Xss256k which is suitable for our use case.
There are however another 285MB non-heap memory (676-389=285).
How can this extra non-heap memory usage be configured ? What is it used
for ?
Below are the relevant node stats.

Regards,
Itai

"process": {
"open_file_descriptors": 340,
"mem": {
"resident_in_bytes": 676884480,
"share_in_bytes": 23248896,
"total_virtual_in_bytes": 1696899072
}
},
"jvm": {
"mem": {
"heap_used_in_bytes": 44794784,
"heap_used_percent": 11,
"heap_committed_in_bytes": 389283840,
"heap_max_in_bytes": 389283840,
"non_heap_used_in_bytes": 44208640,
"non_heap_committed_in_bytes": 44564480,
"pools": {
"young": {
"used_in_bytes": 13765016,
"max_in_bytes": 107479040,
"peak_used_in_bytes": 107479040,
"peak_max_in_bytes": 107479040
},
"survivor": {
"used_in_bytes": 5086896,
"max_in_bytes": 13369344,
"peak_used_in_bytes": 13369344,
"peak_max_in_bytes": 13369344
},
"old": {
"used_in_bytes": 25942872,
"max_in_bytes": 268435456,
"peak_used_in_bytes": 25942872,
"peak_max_in_bytes": 268435456
}
}
},

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/178f4f2f-5dfe-418e-82a3-de505a9ebd9a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Why would you want that?

Locking heap memory usage is done by Elasticsearch on data nodes to reduce
GC rounds, mainly because it loads a lot of data that is best managed by ES
itself.

On client nodes you don't need that (and if you did, you wouldn't be using
that small heap sizes)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Jan 13, 2015 at 12:37 PM, Itai Frenkel itaifrenkel@live.com wrote:

Hello,

We are running a node client on each machine with small JVM heap -Xms384m
-Xmx384m -Xss256k which is suitable for our use case.
There are however another 285MB non-heap memory (676-389=285).
How can this extra non-heap memory usage be configured ? What is it used
for ?
Below are the relevant node stats.

Regards,
Itai

"process": {
"open_file_descriptors": 340,
"mem": {
"resident_in_bytes": 676884480,
"share_in_bytes": 23248896,
"total_virtual_in_bytes": 1696899072
}
},
"jvm": {
"mem": {
"heap_used_in_bytes": 44794784,
"heap_used_percent": 11,
"heap_committed_in_bytes": 389283840,
"heap_max_in_bytes": 389283840,
"non_heap_used_in_bytes": 44208640,
"non_heap_committed_in_bytes": 44564480,
"pools": {
"young": {
"used_in_bytes": 13765016,
"max_in_bytes": 107479040,
"peak_used_in_bytes": 107479040,
"peak_max_in_bytes": 107479040
},
"survivor": {
"used_in_bytes": 5086896,
"max_in_bytes": 13369344,
"peak_used_in_bytes": 13369344,
"peak_max_in_bytes": 13369344
},
"old": {
"used_in_bytes": 25942872,
"max_in_bytes": 268435456,
"peak_used_in_bytes": 25942872,
"peak_max_in_bytes": 268435456
}
}
},

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/178f4f2f-5dfe-418e-82a3-de505a9ebd9a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/178f4f2f-5dfe-418e-82a3-de505a9ebd9a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuWvT7Co6PMJZjYY5tA1XTtiMaBHRZDyynX4zKTmC%3D6Bg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

I would like to reduce the amount of non-heap memory used by a client node.
I would like to reclaim as much as I can from these 280MB, what is it used
for?

On Tuesday, January 13, 2015 at 12:56:32 PM UTC+2, Itamar Syn-Hershko wrote:

Why would you want that?

Locking heap memory usage is done by Elasticsearch on data nodes to reduce
GC rounds, mainly because it loads a lot of data that is best managed by ES
itself.

On client nodes you don't need that (and if you did, you wouldn't be using
that small heap sizes)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Jan 13, 2015 at 12:37 PM, Itai Frenkel <itaif...@live.com
<javascript:>> wrote:

Hello,

We are running a node client on each machine with small JVM heap -Xms384m
-Xmx384m -Xss256k which is suitable for our use case.
There are however another 285MB non-heap memory (676-389=285).
How can this extra non-heap memory usage be configured ? What is it used
for ?
Below are the relevant node stats.

Regards,
Itai

"process": {
"open_file_descriptors": 340,
"mem": {
"resident_in_bytes": 676884480,
"share_in_bytes": 23248896,
"total_virtual_in_bytes": 1696899072
}
},
"jvm": {
"mem": {
"heap_used_in_bytes": 44794784,
"heap_used_percent": 11,
"heap_committed_in_bytes": 389283840,
"heap_max_in_bytes": 389283840,
"non_heap_used_in_bytes": 44208640,
"non_heap_committed_in_bytes": 44564480,
"pools": {
"young": {
"used_in_bytes": 13765016,
"max_in_bytes": 107479040,
"peak_used_in_bytes": 107479040,
"peak_max_in_bytes": 107479040
},
"survivor": {
"used_in_bytes": 5086896,
"max_in_bytes": 13369344,
"peak_used_in_bytes": 13369344,
"peak_max_in_bytes": 13369344
},
"old": {
"used_in_bytes": 25942872,
"max_in_bytes": 268435456,
"peak_used_in_bytes": 25942872,
"peak_max_in_bytes": 268435456
}
}
},

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/178f4f2f-5dfe-418e-82a3-de505a9ebd9a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/178f4f2f-5dfe-418e-82a3-de505a9ebd9a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d287a86-a84d-4e0f-b914-c7013b4d6ce1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

I'd guess that'd be virtual memory, that the OS handles.

On 14 January 2015 at 10:14, Itai Frenkel itaifrenkel@live.com wrote:

I would like to reduce the amount of non-heap memory used by a client
node. I would like to reclaim as much as I can from these 280MB, what is it
used for?

On Tuesday, January 13, 2015 at 12:56:32 PM UTC+2, Itamar Syn-Hershko
wrote:

Why would you want that?

Locking heap memory usage is done by Elasticsearch on data nodes to
reduce GC rounds, mainly because it loads a lot of data that is best
managed by ES itself.

On client nodes you don't need that (and if you did, you wouldn't be
using that small heap sizes)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Tue, Jan 13, 2015 at 12:37 PM, Itai Frenkel itaif...@live.com wrote:

Hello,

We are running a node client on each machine with small JVM
heap -Xms384m -Xmx384m -Xss256k which is suitable for our use case.
There are however another 285MB non-heap memory (676-389=285).
How can this extra non-heap memory usage be configured ? What is it used
for ?
Below are the relevant node stats.

Regards,
Itai

"process": {
"open_file_descriptors": 340,
"mem": {
"resident_in_bytes": 676884480,
"share_in_bytes": 23248896,
"total_virtual_in_bytes": 1696899072
}
},
"jvm": {
"mem": {
"heap_used_in_bytes": 44794784,
"heap_used_percent": 11,
"heap_committed_in_bytes": 389283840,
"heap_max_in_bytes": 389283840,
"non_heap_used_in_bytes": 44208640,
"non_heap_committed_in_bytes": 44564480,
"pools": {
"young": {
"used_in_bytes": 13765016,
"max_in_bytes": 107479040,
"peak_used_in_bytes": 107479040,
"peak_max_in_bytes": 107479040
},
"survivor": {
"used_in_bytes": 5086896,
"max_in_bytes": 13369344,
"peak_used_in_bytes": 13369344,
"peak_max_in_bytes": 13369344
},
"old": {
"used_in_bytes": 25942872,
"max_in_bytes": 268435456,
"peak_used_in_bytes": 25942872,
"peak_max_in_bytes": 268435456
}
}
},

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/178f4f2f-5dfe-418e-82a3-de505a9ebd9a%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/178f4f2f-5dfe-418e-82a3-de505a9ebd9a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/1d287a86-a84d-4e0f-b914-c7013b4d6ce1%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/1d287a86-a84d-4e0f-b914-c7013b4d6ce1%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X_J%2Bm9D7pK4dZTs2_fLFOC4rC3Qw9%3D8s_6eY3tahAcdBg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.