Master only node hardware sizing

Hello,
I'm currently thinking of creating VM nodes for the masters.

Today, several nodes have master and data node roles.
But I have OOM memory errors and so masters crashed frequently.

What would be the correct hardware sizing for a master node only (like 2
CPUs, 4GB RAM) for managing a cluster of 20 data nodes with 4000 shards ?

regards

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I'd suggest you go for 8GB system RAM with a small disk and then also use
these nodes as clients - ie query management.

You may need more RAM, but that should be a good start.

On 11 November 2014 19:35, lagarutte via elasticsearch <
elasticsearch@googlegroups.com> wrote:

Hello,
I'm currently thinking of creating VM nodes for the masters.

Today, several nodes have master and data node roles.
But I have OOM memory errors and so masters crashed frequently.

What would be the correct hardware sizing for a master node only (like 2
CPUs, 4GB RAM) for managing a cluster of 20 data nodes with 4000 shards ?

regards

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmVHuWRz9SOBN8uGvNaz1NM6D5_Z2nEYYrKsWPJDztvvw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Ok, since the master doesn't contain any data, and don't do lot ofs IOs:
For 8GB RAM, what's the recommended HEAP_SIZE ? 7GB

and i don't fully understand the client (query management node).
Today, i have set node.master=true and node.data=true.
and with new master only, i will have masters only nodes and data only
nodes.

What the correct setting for client node ? node.client=true (with
node.master=false and node.data=false) with http adress set ?
so i don't need to tell the masters adress to the other applications like
logstash, tomcat, ... ?

Le mardi 11 novembre 2014 09:45:17 UTC+1, Mark Walkom a écrit :

I'd suggest you go for 8GB system RAM with a small disk and then also use
these nodes as clients - ie query management.

You may need more RAM, but that should be a good start.

On 11 November 2014 19:35, lagarutte via elasticsearch <
elasti...@googlegroups.com <javascript:>> wrote:

Hello,
I'm currently thinking of creating VM nodes for the masters.

Today, several nodes have master and data node roles.
But I have OOM memory errors and so masters crashed frequently.

What would be the correct hardware sizing for a master node only (like 2
CPUs, 4GB RAM) for managing a cluster of 20 data nodes with 4000 shards ?

regards

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2a49b009-4522-45b1-86af-9bafdb56db95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You should use 50% of your system memory for heap.

A client is just a node that is neither a master or a data node, though
once you set a node to master, you can leverage it as a client as well.

On 12 November 2014 05:02, lagarutte via elasticsearch <
elasticsearch@googlegroups.com> wrote:

Ok, since the master doesn't contain any data, and don't do lot ofs IOs:
For 8GB RAM, what's the recommended HEAP_SIZE ? 7GB

and i don't fully understand the client (query management node).
Today, i have set node.master=true and node.data=true.
and with new master only, i will have masters only nodes and data only
nodes.

What the correct setting for client node ? node.client=true (with
node.master=false and node.data=false) with http adress set ?
so i don't need to tell the masters adress to the other applications like
logstash, tomcat, ... ?

Le mardi 11 novembre 2014 09:45:17 UTC+1, Mark Walkom a écrit :

I'd suggest you go for 8GB system RAM with a small disk and then also use
these nodes as clients - ie query management.

You may need more RAM, but that should be a good start.

On 11 November 2014 19:35, lagarutte via elasticsearch <
elasti...@googlegroups.com> wrote:

Hello,
I'm currently thinking of creating VM nodes for the masters.

Today, several nodes have master and data node roles.
But I have OOM memory errors and so masters crashed frequently.

What would be the correct hardware sizing for a master node only (like 2
CPUs, 4GB RAM) for managing a cluster of 20 data nodes with 4000 shards ?

regards

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a49b009-4522-45b1-86af-9bafdb56db95%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2a49b009-4522-45b1-86af-9bafdb56db95%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZ%3D9FR%3DkU49DTV9KBVnCzz1pDRzPju3yPjy6Awtf%3DwAKGQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

if i set 4Gb heap for 8Gb system master node only, for what purposes are
the remaning 4GB ?

For the filesystem cache (but the master don't do IOs) ?

Le mardi 11 novembre 2014 22:46:50 UTC+1, Mark Walkom a écrit :

You should use 50% of your system memory for heap.

A client is just a node that is neither a master or a data node, though
once you set a node to master, you can leverage it as a client as well.

On 12 November 2014 05:02, lagarutte via elasticsearch <
elasti...@googlegroups.com <javascript:>> wrote:

Ok, since the master doesn't contain any data, and don't do lot ofs IOs:
For 8GB RAM, what's the recommended HEAP_SIZE ? 7GB

and i don't fully understand the client (query management node).
Today, i have set node.master=true and node.data=true.
and with new master only, i will have masters only nodes and data only
nodes.

What the correct setting for client node ? node.client=true (with
node.master=false and node.data=false) with http adress set ?
so i don't need to tell the masters adress to the other applications like
logstash, tomcat, ... ?

Le mardi 11 novembre 2014 09:45:17 UTC+1, Mark Walkom a écrit :

I'd suggest you go for 8GB system RAM with a small disk and then also
use these nodes as clients - ie query management.

You may need more RAM, but that should be a good start.

On 11 November 2014 19:35, lagarutte via elasticsearch <
elasti...@googlegroups.com> wrote:

Hello,
I'm currently thinking of creating VM nodes for the masters.

Today, several nodes have master and data node roles.
But I have OOM memory errors and so masters crashed frequently.

What would be the correct hardware sizing for a master node only (like
2 CPUs, 4GB RAM) for managing a cluster of 20 data nodes with 4000 shards ?

regards

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/2a49b009-4522-45b1-86af-9bafdb56db95%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2a49b009-4522-45b1-86af-9bafdb56db95%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0b48c2e8-9e95-46bd-80f7-d99e67980ada%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

That's actually a great point!

You can probably reduce the RAM and then see how it performs.

On 12 November 2014 20:55, lagarutte via elasticsearch <
elasticsearch@googlegroups.com> wrote:

if i set 4Gb heap for 8Gb system master node only, for what purposes are
the remaning 4GB ?

For the filesystem cache (but the master don't do IOs) ?

Le mardi 11 novembre 2014 22:46:50 UTC+1, Mark Walkom a écrit :

You should use 50% of your system memory for heap.

A client is just a node that is neither a master or a data node, though
once you set a node to master, you can leverage it as a client as well.

On 12 November 2014 05:02, lagarutte via elasticsearch <
elasti...@googlegroups.com> wrote:

Ok, since the master doesn't contain any data, and don't do lot ofs IOs:
For 8GB RAM, what's the recommended HEAP_SIZE ? 7GB

and i don't fully understand the client (query management node).
Today, i have set node.master=true and node.data=true.
and with new master only, i will have masters only nodes and data only
nodes.

What the correct setting for client node ? node.client=true (with
node.master=false and node.data=false) with http adress set ?
so i don't need to tell the masters adress to the other applications
like logstash, tomcat, ... ?

Le mardi 11 novembre 2014 09:45:17 UTC+1, Mark Walkom a écrit :

I'd suggest you go for 8GB system RAM with a small disk and then also
use these nodes as clients - ie query management.

You may need more RAM, but that should be a good start.

On 11 November 2014 19:35, lagarutte via elasticsearch <
elasti...@googlegroups.com> wrote:

Hello,
I'm currently thinking of creating VM nodes for the masters.

Today, several nodes have master and data node roles.
But I have OOM memory errors and so masters crashed frequently.

What would be the correct hardware sizing for a master node only (like
2 CPUs, 4GB RAM) for managing a cluster of 20 data nodes with 4000 shards ?

regards

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40goo
glegroups.com
https://groups.google.com/d/msgid/elasticsearch/6447f37c-68d7-4c22-aa1f-4bbf022df3b7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/2a49b009-4522-45b1-86af-9bafdb56db95%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/2a49b009-4522-45b1-86af-9bafdb56db95%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0b48c2e8-9e95-46bd-80f7-d99e67980ada%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/0b48c2e8-9e95-46bd-80f7-d99e67980ada%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZmmGuaQK-5e4zHwWDd5VgibLqOT8i%3Dwm59i0v-wcT16nw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.