Currently I have a 3 node ElasticSearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:
All nodes can be master and are data nodes.
3 shards and 1 replica
6 different indexes
I'm starting to run into issues of ElasticSearch bogging down on searches
and is completely freezing sometimes at night. I've dedicated 9 gig to heap
size and it says i'm using ~60% of the heap RAM and about 70% of the
overall heap. So even though I'm using quite a bit of the heap, I'm not
maxed out. I've attached a screenshot of the exact stats from Elastic HQ.
I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.
My question is what can I do to help the stability and speed of my cluster.
Currently I'm having issues with 1 node going down and it taking everything
else down. The HA portion isn't working very well. I'm debating about
either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.
Basically, the recommended pattern talks about isolating responsibilities.
A node should either be a data node, master-eligible node, or an external
gateway to the cluster (client node)
Currently I have a 3 node Elasticsearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:
All nodes can be master and are data nodes.
3 shards and 1 replica
6 different indexes
I'm starting to run into issues of Elasticsearch bogging down on searches
and is completely freezing sometimes at night. I've dedicated 9 gig to heap
size and it says i'm using ~60% of the heap RAM and about 70% of the
overall heap. So even though I'm using quite a bit of the heap, I'm not
maxed out. I've attached a screenshot of the exact stats from Elastic HQ.
I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.
My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.
Basically, the recommended pattern talks about isolating responsibilities.
A node should either be a data node, master-eligible node, or an external
gateway to the cluster (client node)
Currently I have a 3 node Elasticsearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:
All nodes can be master and are data nodes.
3 shards and 1 replica
6 different indexes
I'm starting to run into issues of Elasticsearch bogging down on searches
and is completely freezing sometimes at night. I've dedicated 9 gig to heap
size and it says i'm using ~60% of the heap RAM and about 70% of the
overall heap. So even though I'm using quite a bit of the heap, I'm not
maxed out. I've attached a screenshot of the exact stats from Elastic HQ.
I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.
My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.
Basically, the recommended pattern talks about isolating
responsibilities. A node should either be a data node, master-eligible
node, or an external gateway to the cluster (client node)
Currently I have a 3 node Elasticsearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:
All nodes can be master and are data nodes.
3 shards and 1 replica
6 different indexes
I'm starting to run into issues of Elasticsearch bogging down on
searches and is completely freezing sometimes at night. I've dedicated 9
gig to heap size and it says i'm using ~60% of the heap RAM and about 70%
of the overall heap. So even though I'm using quite a bit of the heap, I'm
not maxed out. I've attached a screenshot of the exact stats from Elastic
HQ. I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.
My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.
We're in a similar position to the OP, expanding and rearchitecting a
cluster that's currently undersized. We'll have our new cluster spread
across two datacenters, so I'm trying to figure out how to setup the nodes.
We'll have 4 data nodes, the database will probably land somewhere around
750-1000 GB, it's a Graylog2 cluster so I'm not sure how many shards we'll
have in the end, but we'll go with 2 replicas per shard so that we never
end up with one shard being in just one DC. Likewise the Graylog2 servers
will be spread between the two DC's, two per DC.
How many masters would be recommended for this? My initial thought would be
that two would be good enough. The two failure scenarios I see as most
likely(or perhaps least unlikely if you're an optimist) are:
Master ES crashes - Other master takes over.
The link between the DC's goes down or flaps - Won't matter if we have 1 or
2 masters per DC anyway.
So would 2 masters, one per DC be good enough for this setup?
Basically, the recommended pattern talks about isolating
responsibilities. A node should either be a data node, master-eligible
node, or an external gateway to the cluster (client node)
On Thu, Feb 12, 2015 at 4:08 PM, Eric <eric.l...@gmail.com <javascript:>>
wrote:
Hello,
Currently I have a 3 node Elasticsearch cluster. Each node is a RHEL VM
with 16 gig RAM. The basic config is:
All nodes can be master and are data nodes.
3 shards and 1 replica
6 different indexes
I'm starting to run into issues of Elasticsearch bogging down on
searches and is completely freezing sometimes at night. I've dedicated 9
gig to heap size and it says i'm using ~60% of the heap RAM and about 70%
of the overall heap. So even though I'm using quite a bit of the heap, I'm
not maxed out. I've attached a screenshot of the exact stats from Elastic
HQ. I'm averaging around 10,000 events/sec coming into the cluster from 6
different Logstash instances on another server.
My question is what can I do to help the stability and speed of my
cluster. Currently I'm having issues with 1 node going down and it taking
everything else down. The HA portion isn't working very well. I'm debating
about either adding 1 more node with the exact same stats or adding 2 more
smaller VMs that will act as master nodes only. I didn't know which one was
recommended or where I would get the biggest bang for the buck.
We're in a similar position to the OP, expanding and rearchitecting a
cluster that's currently undersized. We'll have our new cluster spread
across two datacenters, so I'm trying to figure out how to setup the nodes.
We'll have 4 data nodes, the database will probably land somewhere around
750-1000 GB, it's a Graylog2 cluster so I'm not sure how many shards we'll
have in the end, but we'll go with 2 replicas per shard so that we never
end up with one shard being in just one DC. Likewise the Graylog2 servers
will be spread between the two DC's, two per DC.
How many masters would be recommended for this? My initial thought would
be that two would be good enough. The two failure scenarios I see as most
likely(or perhaps least unlikely if you're an optimist) are:
Master ES crashes - Other master takes over.
The link between the DC's goes down or flaps - Won't matter if we have 1
or 2 masters per DC anyway.
So would 2 masters, one per DC be good enough for this setup?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
