The capacity of the elasticsearch nodes have come into question and this
has raised a few questions regards spec of a new node.
Here is the current specs of the elasticsearch nodes ( i have 2 of these):
· HP DL360p Gen8 10-SFF CTO Server
· 64G PC3L-12800R-11
· 8x 900G 10K SAS
· 2x 300G 10K SAS
· 2G FBWC
· Dual 750W PSU
· 4P 1GBE 331FLR
So i am currently running the OS from the 2x 300Gb in a mirror raid, and 2
data logical data drives using 4 of the 8 900GB drives (so 2 striped raids
containing 4x900GB drives).
This is all working fine but the data capacity has become an issue (14TB
total available) . I think for the moment i have enough compute power but
what would happen if i added a lower spec node (or multiple), marked them
as a data node (non-master) but with different storage capacity available.
Say for example 20TB in each.
If the original 2 nodes filed there data stores and the only store
available was the new node(s) then they would be processing the shards
alone and there would be no protection from replica's if this node went
down, no?
As an additional but not as critical at the moment:
If i eventually have the same issue with compute power, if these 2 someday
become saturated and i make another node a master node but it was half the
spec, would elasticsearch realize this and distribute the load or is this
purely down to nginx load distribution?
Can you elaborate what you mean by becoming an issue?
When you add a node into the cluster it will automatically start to
reallocate shards to the new node, you can't have a node sitting there idle
and with lots of disk space free waiting for the other nodes to fill up
before being called upon.
As for nginx, it depends what you do with it. ES will spread the load
amongst the cluster automatically, but if you're using nginx as a front end
then it's up to you to factor the other nodes in.
The capacity of the elasticsearch nodes have come into question and this
has raised a few questions regards spec of a new node.
Here is the current specs of the elasticsearch nodes ( i have 2 of these):
· HP DL360p Gen8 10-SFF CTO Server
· 64G PC3L-12800R-11
· 8x 900G 10K SAS
· 2x 300G 10K SAS
· 2G FBWC
· Dual 750W PSU
· 4P 1GBE 331FLR
So i am currently running the OS from the 2x 300Gb in a mirror raid, and 2
data logical data drives using 4 of the 8 900GB drives (so 2 striped raids
containing 4x900GB drives).
This is all working fine but the data capacity has become an issue (14TB
total available) . I think for the moment i have enough compute power but
what would happen if i added a lower spec node (or multiple), marked them
as a data node (non-master) but with different storage capacity available.
Say for example 20TB in each.
If the original 2 nodes filed there data stores and the only store
available was the new node(s) then they would be processing the shards
alone and there would be no protection from replica's if this node went
down, no?
As an additional but not as critical at the moment:
If i eventually have the same issue with compute power, if these 2 someday
become saturated and i make another node a master node but it was half the
spec, would elasticsearch realize this and distribute the load or is this
purely down to nginx load distribution?
It seems that some of the systems are going to be generating more logs than
we previously thought (approx 250GB a day) so 14TB won't last too long. So
we are looking into new nodes.
I think we currently have enough compute power (won't know until this goes
live) so the current nodes can be kept as masters.
So what would happen if say we added a 20TB data non-master node when the
storage use gets to 21TB(distributed over the 3 nodes so 7TB on each making
the first 2 nodes full)?
Would it then only store the data\replicas\shards on the new 3rd node as
that is the only one with space?
Thanks
On Monday, 5 January 2015 20:53:28 UTC, Mark Walkom wrote:
Can you elaborate what you mean by becoming an issue?
When you add a node into the cluster it will automatically start to
reallocate shards to the new node, you can't have a node sitting there idle
and with lots of disk space free waiting for the other nodes to fill up
before being called upon.
As for nginx, it depends what you do with it. ES will spread the load
amongst the cluster automatically, but if you're using nginx as a front end
then it's up to you to factor the other nodes in.
On 6 January 2015 at 01:51, Simon Thorley <si...@thenom.co.uk
<javascript:>> wrote:
Hi all,
I have designed and setup an ELK stack that can be expanded horizontally
(i hope) using the following technologies:
The capacity of the elasticsearch nodes have come into question and this
has raised a few questions regards spec of a new node.
Here is the current specs of the elasticsearch nodes ( i have 2 of these):
· HP DL360p Gen8 10-SFF CTO Server
· 64G PC3L-12800R-11
· 8x 900G 10K SAS
· 2x 300G 10K SAS
· 2G FBWC
· Dual 750W PSU
· 4P 1GBE 331FLR
So i am currently running the OS from the 2x 300Gb in a mirror raid, and
2 data logical data drives using 4 of the 8 900GB drives (so 2 striped
raids containing 4x900GB drives).
This is all working fine but the data capacity has become an issue (14TB
total available) . I think for the moment i have enough compute power but
what would happen if i added a lower spec node (or multiple), marked them
as a data node (non-master) but with different storage capacity available.
Say for example 20TB in each.
If the original 2 nodes filed there data stores and the only store
available was the new node(s) then they would be processing the shards
alone and there would be no protection from replica's if this node went
down, no?
As an additional but not as critical at the moment:
If i eventually have the same issue with compute power, if these 2
someday become saturated and i make another node a master node but it was
half the spec, would elasticsearch realize this and distribute the load
or is this purely down to nginx load distribution?
It seems that some of the systems are going to be generating more logs
than we previously thought (approx 250GB a day) so 14TB won't last too
long. So we are looking into new nodes.
I think we currently have enough compute power (won't know until this goes
live) so the current nodes can be kept as masters.
So what would happen if say we added a 20TB data non-master node when the
storage use gets to 21TB(distributed over the 3 nodes so 7TB on each making
the first 2 nodes full)?
Would it then only store the data\replicas\shards on the new 3rd node as
that is the only one with space?
Thanks
On Monday, 5 January 2015 20:53:28 UTC, Mark Walkom wrote:
Can you elaborate what you mean by becoming an issue?
When you add a node into the cluster it will automatically start to
reallocate shards to the new node, you can't have a node sitting there idle
and with lots of disk space free waiting for the other nodes to fill up
before being called upon.
As for nginx, it depends what you do with it. ES will spread the load
amongst the cluster automatically, but if you're using nginx as a front end
then it's up to you to factor the other nodes in.
The capacity of the elasticsearch nodes have come into question and this
has raised a few questions regards spec of a new node.
Here is the current specs of the elasticsearch nodes ( i have 2 of
these):
· HP DL360p Gen8 10-SFF CTO Server
· 64G PC3L-12800R-11
· 8x 900G 10K SAS
· 2x 300G 10K SAS
· 2G FBWC
· Dual 750W PSU
· 4P 1GBE 331FLR
So i am currently running the OS from the 2x 300Gb in a mirror raid, and
2 data logical data drives using 4 of the 8 900GB drives (so 2 striped
raids containing 4x900GB drives).
This is all working fine but the data capacity has become an issue (14TB
total available) . I think for the moment i have enough compute power but
what would happen if i added a lower spec node (or multiple), marked them
as a data node (non-master) but with different storage capacity available.
Say for example 20TB in each.
If the original 2 nodes filed there data stores and the only store
available was the new node(s) then they would be processing the shards
alone and there would be no protection from replica's if this node went
down, no?
As an additional but not as critical at the moment:
If i eventually have the same issue with compute power, if these 2
someday become saturated and i make another node a master node but it was
half the spec, would elasticsearch realize this and distribute the load
or is this purely down to nginx load distribution?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.