Master Node vs. Data Node Architecture

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

  • All nodes can be master and are data nodes.
  • 3 shards and 1 replica
  • 6 different indexes

I'm starting to run into issues of ElasticSearch bogging down on searches
and is completely freezing sometimes at night. I've dedicated 9 gig to heap
size and it says i'm using ~60% of the heap RAM and about 70% of the
overall heap. So even though I'm using quite a bit of the heap, I'm not
maxed out. I've attached a screenshot of the exact stats from Elastic HQ.
I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.

My question is what can I do to help the stability and speed of my cluster.
Currently I'm having issues with 1 node going down and it taking everything
else down. The HA portion isn't working very well. I'm debating about
either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

See this:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html

Basically, the recommended pattern talks about isolating responsibilities.
A node should either be a data node, master-eligible node, or an external
gateway to the cluster (client node)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:08 PM, Eric eric.luellen@gmail.com wrote:

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

  • All nodes can be master and are data nodes.
  • 3 shards and 1 replica
  • 6 different indexes

I'm starting to run into issues of ElasticSearch bogging down on searches
and is completely freezing sometimes at night. I've dedicated 9 gig to heap
size and it says i'm using ~60% of the heap RAM and about 70% of the
overall heap. So even though I'm using quite a bit of the heap, I'm not
maxed out. I've attached a screenshot of the exact stats from Elastic HQ.
I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.

My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Except that is overkill when you only have 3 nodes.

How much data do you have in the cluster?

On 13 February 2015 at 01:15, Itamar Syn-Hershko itamar@code972.com wrote:

See this:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html

Basically, the recommended pattern talks about isolating responsibilities.
A node should either be a data node, master-eligible node, or an external
gateway to the cluster (client node)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:08 PM, Eric eric.luellen@gmail.com wrote:

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

  • All nodes can be master and are data nodes.
  • 3 shards and 1 replica
  • 6 different indexes

I'm starting to run into issues of ElasticSearch bogging down on searches
and is completely freezing sometimes at night. I've dedicated 9 gig to heap
size and it says i'm using ~60% of the heap RAM and about 70% of the
overall heap. So even though I'm using quite a bit of the heap, I'm not
maxed out. I've attached a screenshot of the exact stats from Elastic HQ.
I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.

My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X93EwqeGf9S4UpMvtJy3%2BmaAjovfVicj7LRHz%2BPyAbSug%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Depending why the node goes down - going mid-way with dedicated master
nodes is sometimes the solution

And if this is due to massive use of aggregations, doc-values may be the
answer (or larger heap, but that's costlier)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 11:40 PM, Mark Walkom markwalkom@gmail.com wrote:

Except that is overkill when you only have 3 nodes.

How much data do you have in the cluster?

On 13 February 2015 at 01:15, Itamar Syn-Hershko itamar@code972.com
wrote:

See this:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html

Basically, the recommended pattern talks about isolating
responsibilities. A node should either be a data node, master-eligible
node, or an external gateway to the cluster (client node)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:08 PM, Eric eric.luellen@gmail.com wrote:

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

  • All nodes can be master and are data nodes.
  • 3 shards and 1 replica
  • 6 different indexes

I'm starting to run into issues of ElasticSearch bogging down on
searches and is completely freezing sometimes at night. I've dedicated 9
gig to heap size and it says i'm using ~60% of the heap RAM and about 70%
of the overall heap. So even though I'm using quite a bit of the heap, I'm
not maxed out. I've attached a screenshot of the exact stats from Elastic
HQ. I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.

My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X93EwqeGf9S4UpMvtJy3%2BmaAjovfVicj7LRHz%2BPyAbSug%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X93EwqeGf9S4UpMvtJy3%2BmaAjovfVicj7LRHz%2BPyAbSug%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZvFX-bEqpSEnh3EmdbzAwAhDLE7PYDivd5Q2VnFu_xviA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I might as well ask a related question...

We're in a similar position to the OP, expanding and rearchitecting a
cluster that's currently undersized. We'll have our new cluster spread
across two datacenters, so I'm trying to figure out how to setup the nodes.

We'll have 4 data nodes, the database will probably land somewhere around
750-1000 GB, it's a Graylog2 cluster so I'm not sure how many shards we'll
have in the end, but we'll go with 2 replicas per shard so that we never
end up with one shard being in just one DC. Likewise the Graylog2 servers
will be spread between the two DC's, two per DC.

How many masters would be recommended for this? My initial thought would be
that two would be good enough. The two failure scenarios I see as most
likely(or perhaps least unlikely if you're an optimist) are:
Master ES crashes - Other master takes over.
The link between the DC's goes down or flaps - Won't matter if we have 1 or
2 masters per DC anyway.

So would 2 masters, one per DC be good enough for this setup?

Regards
Johan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e8f9926e-2265-4682-84f7-f1e063781c16%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

According to Elastic HQ I currently am keeping 12 days of logs

  • 8 nodes (3 data nodes that can all be master and 5 Logstash)
  • 399 shards
  • 59 indices
  • 1,540,343,998 documents
  • 380 GB

On Thursday, February 12, 2015 at 4:41:01 PM UTC-5, Mark Walkom wrote:

Except that is overkill when you only have 3 nodes.

How much data do you have in the cluster?

On 13 February 2015 at 01:15, Itamar Syn-Hershko <ita...@code972.com
<javascript:>> wrote:

See this:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/modules-node.html

Basically, the recommended pattern talks about isolating
responsibilities. A node should either be a data node, master-eligible
node, or an external gateway to the cluster (client node)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Thu, Feb 12, 2015 at 4:08 PM, Eric <eric.l...@gmail.com <javascript:>>
wrote:

Hello,

Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:

  • All nodes can be master and are data nodes.
  • 3 shards and 1 replica
  • 6 different indexes

I'm starting to run into issues of ElasticSearch bogging down on
searches and is completely freezing sometimes at night. I've dedicated 9
gig to heap size and it says i'm using ~60% of the heap RAM and about 70%
of the overall heap. So even though I'm using quite a bit of the heap, I'm
not maxed out. I've attached a screenshot of the exact stats from Elastic
HQ. I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.

My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.

Any information would be greatly appreciated.

Thanks,
Eric

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/354a2326-5532-4239-87ea-f02af64fe71f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZureLROJMaO7gVprFjA2OmRZA0ZYyH1v%2Bges06u_V__6w%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/10133d13-ff4d-4798-80e2-4fffabfeb53c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You should really start your own thread for this (please) :slight_smile:

On 13 February 2015 at 19:52, sunner@sunner.com wrote:

I might as well ask a related question...

We're in a similar position to the OP, expanding and rearchitecting a
cluster that's currently undersized. We'll have our new cluster spread
across two datacenters, so I'm trying to figure out how to setup the nodes.

We'll have 4 data nodes, the database will probably land somewhere around
750-1000 GB, it's a Graylog2 cluster so I'm not sure how many shards we'll
have in the end, but we'll go with 2 replicas per shard so that we never
end up with one shard being in just one DC. Likewise the Graylog2 servers
will be spread between the two DC's, two per DC.

How many masters would be recommended for this? My initial thought would
be that two would be good enough. The two failure scenarios I see as most
likely(or perhaps least unlikely if you're an optimist) are:
Master ES crashes - Other master takes over.
The link between the DC's goes down or flaps - Won't matter if we have 1
or 2 masters per DC anyway.

So would 2 masters, one per DC be good enough for this setup?

Regards
Johan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e8f9926e-2265-4682-84f7-f1e063781c16%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e8f9926e-2265-4682-84f7-f1e063781c16%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8e3X0P%2BuQHa280v0GW3ecKwTtJHH9XuCp4X2QNHwTTEA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.