Should ES_HEAP_SIZE be less than 31G?

Elastic Stack Elasticsearch
1

I read somewhere that the ES_HEAP_SIZE is best to be less than 31G, in
this case JVM can use an 32 bit number to address memory locations. If
my server have about 64G ram. This seems a perfect. But what If I have a
server with 128G RAM and sufficient CPU's. Which configuration is better
a) two nodes on server each with ES_HEAP_SIZE say about 31G. b) one node
on the server with ES_HEAP_SIZE say 64G.
Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi ,

I am not sure if there is anything related to ES that makes a high RAM
problematic , but in general very high memory is not advised in java
systems. One of the reasons being the cobra effect which stems from the
fact that garbage collection might need more time for higher memory.

COBRA EFFECT -
https://plumbr.eu/blog/increasing-heap-size-beware-of-the-cobra-effect

Thanks
Vineeth

On Wed, Sep 3, 2014 at 10:04 PM, Jinyuan Zhou zhou.jinyuan@gmail.com
wrote:

I read somewhere that the ES_HEAP_SIZE is best to be less than 31G, in
this case JVM can use an 32 bit number to address memory locations. If
my server have about 64G ram. This seems a perfect. But what If I have a
server with 128G RAM and sufficient CPU's. Which configuration is better
a) two nodes on server each with ES_HEAP_SIZE say about 31G. b) one node
on the server with ES_HEAP_SIZE say 64G.
Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5noOXMuVBZLoz9dJjxko5z4LmWBdk6aSTY2W2sUW7MbuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Yeah. I'd prefer 2 nodes on the same machine but using that: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-cluster.html#allocation-awareness
So primaries and replicas are not allocated on the same machine.

You could also add a 3rd node (master only - 512Mb/1Gb max)

Best

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 3 septembre 2014 à 18:40:27, vineeth mohan (vm.vineethmohan@gmail.com) a écrit:

Hi ,

I am not sure if there is anything related to ES that makes a high RAM problematic , but in general very high memory is not advised in java systems. One of the reasons being the cobra effect which stems from the fact that garbage collection might need more time for higher memory.

COBRA EFFECT - https://plumbr.eu/blog/increasing-heap-size-beware-of-the-cobra-effect

Thanks
Vineeth

On Wed, Sep 3, 2014 at 10:04 PM, Jinyuan Zhou zhou.jinyuan@gmail.com wrote:
I read somewhere that the ES_HEAP_SIZE is best to be less than 31G, in this case JVM can use an 32 bit number to address memory locations. If my server have about 64G ram. This seems a perfect. But what If I have a server with 128G RAM and sufficient CPU's. Which configuration is better a) two nodes on server each with ES_HEAP_SIZE say about 31G. b) one node on the server with ES_HEAP_SIZE say 64G.
Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5noOXMuVBZLoz9dJjxko5z4LmWBdk6aSTY2W2sUW7MbuA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.540746c4.333ab105.ec11%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

4

The actual limitation in Java is compressed pointers:

http://docs.oracle.com/javase/7/docs/technotes/guides/vm/performance-enhancements-7.html#compressedOop

Thankfully Elasticsearch can run multiple nodes on the same server. Just
pay attention to the direct size (off heap memory), mlockall settings and
Lucene merge settings, as well as the allocation property referenced by
David. Elasticsearch scales quite nicely vertically, so I would explore
that option if possible.

Cheers,

Ivan

On Wed, Sep 3, 2014 at 9:40 AM, vineeth mohan vm.vineethmohan@gmail.com
wrote:

Hi ,

I am not sure if there is anything related to ES that makes a high RAM
problematic , but in general very high memory is not advised in java
systems. One of the reasons being the cobra effect which stems from the
fact that garbage collection might need more time for higher memory.

COBRA EFFECT -
https://plumbr.eu/blog/increasing-heap-size-beware-of-the-cobra-effect

Thanks
Vineeth

On Wed, Sep 3, 2014 at 10:04 PM, Jinyuan Zhou zhou.jinyuan@gmail.com
wrote:

I read somewhere that the ES_HEAP_SIZE is best to be less than 31G, in
this case JVM can use an 32 bit number to address memory locations. If
my server have about 64G ram. This seems a perfect. But what If I have a
server with 128G RAM and sufficient CPU's. Which configuration is better
a) two nodes on server each with ES_HEAP_SIZE say about 31G. b) one node
on the server with ES_HEAP_SIZE say 64G.
Thanks,

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5noOXMuVBZLoz9dJjxko5z4LmWBdk6aSTY2W2sUW7MbuA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5noOXMuVBZLoz9dJjxko5z4LmWBdk6aSTY2W2sUW7MbuA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCF25cqzkhZ%2B2ooPfjFMyiSonWdKN%2BfPaiJvkEimK7Y0Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Thanks for answering my questions. I also followed all these interesting
reading and learned a lot.
Cheers,
Jack

Jinyuan (Jack) Zhou

On Wed, Sep 3, 2014 at 10:02 AM, Ivan Brusic ivan@brusic.com wrote:

The actual limitation in Java is compressed pointers:

Java ™ HotSpot Virtual Machine Performance Enhancements

Thankfully Elasticsearch can run multiple nodes on the same server. Just
pay attention to the direct size (off heap memory), mlockall settings and
Lucene merge settings, as well as the allocation property referenced by
David. Elasticsearch scales quite nicely vertically, so I would explore
that option if possible.

Cheers,

Ivan

On Wed, Sep 3, 2014 at 9:40 AM, vineeth mohan vm.vineethmohan@gmail.com
wrote:

Hi ,

I am not sure if there is anything related to ES that makes a high RAM
problematic , but in general very high memory is not advised in java
systems. One of the reasons being the cobra effect which stems from the
fact that garbage collection might need more time for higher memory.

COBRA EFFECT -
https://plumbr.eu/blog/increasing-heap-size-beware-of-the-cobra-effect

Thanks
Vineeth

On Wed, Sep 3, 2014 at 10:04 PM, Jinyuan Zhou zhou.jinyuan@gmail.com
wrote:

I read somewhere that the ES_HEAP_SIZE is best to be less than 31G, in
this case JVM can use an 32 bit number to address memory locations. If
my server have about 64G ram. This seems a perfect. But what If I have a
server with 128G RAM and sufficient CPU's. Which configuration is better
a) two nodes on server each with ES_HEAP_SIZE say about 31G. b) one node
on the server with ES_HEAP_SIZE say 64G.
Thanks,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5noOXMuVBZLoz9dJjxko5z4LmWBdk6aSTY2W2sUW7MbuA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5noOXMuVBZLoz9dJjxko5z4LmWBdk6aSTY2W2sUW7MbuA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qsWdzyqclMc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCF25cqzkhZ%2B2ooPfjFMyiSonWdKN%2BfPaiJvkEimK7Y0Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCF25cqzkhZ%2B2ooPfjFMyiSonWdKN%2BfPaiJvkEimK7Y0Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANBTPCFjoR9Y%3Dg_Aqwd4RsbgTnefrp_yNcf36kZp82nY6Am9JA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

There are certain scalability issues with the garbage collection in the JVM
so if you want to run large heaps with low latency you must expect extra
challenges regarding stability and performance.

For example, G1 GC is a garbage collector that works better than standard
CMS GC at large heaps > 20 GB. For heaps larger than 100 GB and <10ms pause
times, you can not expect much with the current Oracle JVM. There is
Project Shenandoah for minimizing the stop-of-the-world stalls for such
large heaps: JEP 189: Shenandoah: A Low-Pause-Time Garbage Collector (Experimental)

ES scales best over multiple machines horizontally, not vertically. More
RAM does not automatically mean better performance at linear scale at a
certain point - it depends on the JVM if it can keep up.

I do not recommend running more than one JVM per machine, not because of
fault tolerance, but just because of the overhead. The OS kernel must map
the virtual memory (file system cache) and the thread scheduler to more
than one VM and that steals quite few resources under heavy load.

Jörg

On Wed, Sep 3, 2014 at 7:43 PM, Jinyuan Zhou zhou.jinyuan@gmail.com wrote:

Thanks for answering my questions. I also followed all these interesting
reading and learned a lot.
Cheers,
Jack

Jinyuan (Jack) Zhou

On Wed, Sep 3, 2014 at 10:02 AM, Ivan Brusic ivan@brusic.com wrote:

The actual limitation in Java is compressed pointers:

Java ™ HotSpot Virtual Machine Performance Enhancements

Thankfully Elasticsearch can run multiple nodes on the same server. Just
pay attention to the direct size (off heap memory), mlockall settings and
Lucene merge settings, as well as the allocation property referenced by
David. Elasticsearch scales quite nicely vertically, so I would explore
that option if possible.

Cheers,

Ivan

On Wed, Sep 3, 2014 at 9:40 AM, vineeth mohan vm.vineethmohan@gmail.com
wrote:

Hi ,

I am not sure if there is anything related to ES that makes a high RAM
problematic , but in general very high memory is not advised in java
systems. One of the reasons being the cobra effect which stems from the
fact that garbage collection might need more time for higher memory.

COBRA EFFECT -
https://plumbr.eu/blog/increasing-heap-size-beware-of-the-cobra-effect

Thanks
Vineeth

On Wed, Sep 3, 2014 at 10:04 PM, Jinyuan Zhou zhou.jinyuan@gmail.com
wrote:

I read somewhere that the ES_HEAP_SIZE is best to be less than 31G, in
this case JVM can use an 32 bit number to address memory locations. If
my server have about 64G ram. This seems a perfect. But what If I have a
server with 128G RAM and sufficient CPU's. Which configuration is better
a) two nodes on server each with ES_HEAP_SIZE say about 31G. b) one node
on the server with ES_HEAP_SIZE say 64G.
Thanks,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/03180561-2e7d-4e04-ab97-9d8b3a922f86%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5noOXMuVBZLoz9dJjxko5z4LmWBdk6aSTY2W2sUW7MbuA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAGdPd5noOXMuVBZLoz9dJjxko5z4LmWBdk6aSTY2W2sUW7MbuA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qsWdzyqclMc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCF25cqzkhZ%2B2ooPfjFMyiSonWdKN%2BfPaiJvkEimK7Y0Q%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQCF25cqzkhZ%2B2ooPfjFMyiSonWdKN%2BfPaiJvkEimK7Y0Q%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CANBTPCFjoR9Y%3Dg_Aqwd4RsbgTnefrp_yNcf36kZp82nY6Am9JA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CANBTPCFjoR9Y%3Dg_Aqwd4RsbgTnefrp_yNcf36kZp82nY6Am9JA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHnCaeYJp4%2BJKkViFRKQjANTkt_Q%3Dxb_U7QihYRb3fjrQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

On Wed, Sep 3, 2014 at 11:47 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

ES scales best over multiple machines horizontally, not vertically. More
RAM does not automatically mean better performance at linear scale at a
certain point - it depends on the JVM if it can keep up.

Ouch, what a brainfart I had! Yes, I meant to say horizontally as well. If
you shard your indices correctly, or use time-based indices, you can always
increase capacity at a later point. The shard allocator does not take into
consideration the difference in computing power between machines, so it is
best to have all nodes be relatively the same (unless you do manual
allocation). Most clustered software such as Hadoop work in such as fashion.

--
Ivan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDfLn_Zc-VJ4fakTJNS4%2Bur9k9K4ymeJC8CWjZ1NrHbvg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

8

Say if I have two boxes for data node in the cluster: A and B. However, A
is doubles capacity of B. Now I can start two data nodes on A and 1 on B.
With the help of rack id attribute, ES will make sure replicate and its
primary are not located on the same box. In this case, It seems to me that
Machine A will get more shards than Machine B.

Jinyuan (Jack) Zhou

On Thu, Sep 4, 2014 at 9:24 AM, Ivan Brusic ivan@brusic.com wrote:

On Wed, Sep 3, 2014 at 11:47 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

ES scales best over multiple machines horizontally, not vertically. More
RAM does not automatically mean better performance at linear scale at a
certain point - it depends on the JVM if it can keep up.

Ouch, what a brainfart I had! Yes, I meant to say horizontally as well. If
you shard your indices correctly, or use time-based indices, you can always
increase capacity at a later point. The shard allocator does not take into
consideration the difference in computing power between machines, so it is
best to have all nodes be relatively the same (unless you do manual
allocation). Most clustered software such as Hadoop work in such as fashion.

--
Ivan

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/qsWdzyqclMc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDfLn_Zc-VJ4fakTJNS4%2Bur9k9K4ymeJC8CWjZ1NrHbvg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDfLn_Zc-VJ4fakTJNS4%2Bur9k9K4ymeJC8CWjZ1NrHbvg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CANBTPCG2Y0YrG5dsc7DZm%2BHVxRO4PxEa6jWaXUsrLRTFd1Rw4Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.