Managing ES servers with differing data disk sizes

Elastic Stack Elasticsearch
1

Hello ES users,

I have a question about how to manage an ES cluster where the servers have
different sized data disks. I have 4 servers with 1 x 1TB disks for ES
data, and I am planning on adding an additional 2 servers that will have 4
x 4TB disks for ES data.

Is ES "aware" of the capacity of its data disks at all and do anything to
balance the load evenly, or will that fall to me to manually move shards
around to do that?

Thanks for the input :slight_smile:
Chris

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAND3DpgCrPrkKTjV29i%3Dm9Jg8g_%2BvgoO6Ai7q4WZOuUbJcbvpg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

2

Since version 1.3.0, yes: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-allocation.html#disk

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet | @elasticsearchfr

Le 29 juillet 2014 à 18:12:24, Chris Neal (chris.neal@derbysoft.net) a écrit:

Hello ES users,

I have a question about how to manage an ES cluster where the servers have different sized data disks. I have 4 servers with 1 x 1TB disks for ES data, and I am planning on adding an additional 2 servers that will have 4 x 4TB disks for ES data.

Is ES "aware" of the capacity of its data disks at all and do anything to balance the load evenly, or will that fall to me to manually move shards around to do that?

Thanks for the input :slight_smile:
Chris

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAND3DpgCrPrkKTjV29i%3Dm9Jg8g_%2BvgoO6Ai7q4WZOuUbJcbvpg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.53d7c851.12200854.f0d0%40MacBook-Air-de-David.local.
For more options, visit https://groups.google.com/d/optout.

3

On Tue, Jul 29, 2014 at 12:12 PM, Chris Neal chris.neal@derbysoft.net
wrote:

Hello ES users,

I have a question about how to manage an ES cluster where the servers have
different sized data disks. I have 4 servers with 1 x 1TB disks for ES
data, and I am planning on adding an additional 2 servers that will have 4
x 4TB disks for ES data.

Is ES "aware" of the capacity of its data disks at all and do anything to
balance the load evenly, or will that fall to me to manually move shards
around to do that?

It'll make sure it doesn't fill up the disks if you enable the disk
allocaiton decider (enabled by default from 1.3 onwards) but beyond it
won't do anything like balance shards proportionally to disk capacity. In
many cases that wouldn't be the right thing to do anyway. You can use the
allocation awareness api stick shards to some machines or force them away
from others but there is no way to say "this node is twice as good put
twice as many shards on it" without running two copies of Elasticsearch on
the node.

Thanks for the input :slight_smile:
Chris

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAND3DpgCrPrkKTjV29i%3Dm9Jg8g_%2BvgoO6Ai7q4WZOuUbJcbvpg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAND3DpgCrPrkKTjV29i%3Dm9Jg8g_%2BvgoO6Ai7q4WZOuUbJcbvpg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2WzPFRZAQSBZnVkVp%3DgppX5JGf%2BqqFUfPpQ1_tTQ-hQA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Thanks very much guys for your input. I'm in the process of upgrading from
1.2.2 to 1.3.1 now, so I'll read up on the Index Shard Allocation link!

Chris

On Tue, Jul 29, 2014 at 11:16 AM, Nikolas Everett nik9000@gmail.com wrote:

On Tue, Jul 29, 2014 at 12:12 PM, Chris Neal chris.neal@derbysoft.net
wrote:

Hello ES users,

I have a question about how to manage an ES cluster where the servers
have different sized data disks. I have 4 servers with 1 x 1TB disks for
ES data, and I am planning on adding an additional 2 servers that will have
4 x 4TB disks for ES data.

Is ES "aware" of the capacity of its data disks at all and do anything to
balance the load evenly, or will that fall to me to manually move shards
around to do that?

It'll make sure it doesn't fill up the disks if you enable the disk
allocaiton decider (enabled by default from 1.3 onwards) but beyond it
won't do anything like balance shards proportionally to disk capacity. In
many cases that wouldn't be the right thing to do anyway. You can use the
allocation awareness api stick shards to some machines or force them away
from others but there is no way to say "this node is twice as good put
twice as many shards on it" without running two copies of Elasticsearch on
the node.

Thanks for the input :slight_smile:
Chris

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAND3DpgCrPrkKTjV29i%3Dm9Jg8g_%2BvgoO6Ai7q4WZOuUbJcbvpg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAND3DpgCrPrkKTjV29i%3Dm9Jg8g_%2BvgoO6Ai7q4WZOuUbJcbvpg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2WzPFRZAQSBZnVkVp%3DgppX5JGf%2BqqFUfPpQ1_tTQ-hQA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2WzPFRZAQSBZnVkVp%3DgppX5JGf%2BqqFUfPpQ1_tTQ-hQA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAND3Dpi6QRi4rtViwdrQw50GfUWAGepjivxHv7BqavqG9-Rv0g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.