Shard Allocation Awareness in ec2

Elastic Stack Elasticsearch
1

Hi,

I'm setting up elastic search cluster in ec2

This is the discovery configuration

discovery:
        type:   ec2
        ec2:
                groups:                 elasticsearch
                host_type:              private_ip
                availability_zones:     us-east-1a,us-east-1c,us-
east-1d
                any_group:              false
                ping_timeout:           5
                tag:
                        stage:  dev

Now

From the documentation it's not clear if Cluster module allocation
awareness makes use of the fact that node is running in ec2 - which
seems natural to do.

So the question is do I need to set routing.allocation.awareness
explicitly like this:

cluster:
name: dev
routing.allocation.awareness:
attributes:
availability_zone
force.availability_zone.values: us-east-1a,us-
east-1c,us-east-1d

node:
availability_zone: us-east-1a

Or this could be simplified or completely omitted with same effect?
Like specifying node.availability_zone - seems to be unneeded - as we
already know such info from the ec2 discovery module.

Thanks
Alex

2

Yes, you will need to explicitly set the awareness in the routing
allocation forced part. You can set cloud.node.auto_attributes to true and
it will automatically add the ec2 relevant (node.aws_availability_zone) to
the node attributes.

On Wed, Dec 14, 2011 at 8:26 AM, Piavlo lolitushka@gmail.com wrote:

Hi,

I'm setting up Elasticsearch cluster in ec2

This is the discovery configuration

discovery:
type: ec2
ec2:
groups: elasticsearch
host_type: private_ip
availability_zones: us-east-1a,us-east-1c,us-
east-1d
any_group: false
ping_timeout: 5
tag:
stage: dev

Now

From the documentation it's not clear if Cluster module allocation
awareness makes use of the fact that node is running in ec2 - which
seems natural to do.

So the question is do I need to set routing.allocation.awareness
explicitly like this:

cluster:
name: dev
routing.allocation.awareness:
attributes:
availability_zone
force.availability_zone.values: us-east-1a,us-
east-1c,us-east-1d

node:
availability_zone: us-east-1a

Or this could be simplified or completely omitted with same effect?
Like specifying node.availability_zone - seems to be unneeded - as we
already know such info from the ec2 discovery module.

Thanks
Alex

3

Hi Shay

Thanks for the rpley

To be sure I understood you correctly - could you please confirm that
the following is a full correct config to do the job?

cluster:
name: dev
routing.allocation.awareness:
attributes:
aws_availability_zone
force.aws_availability_zone.values: us-east-1a,us-
east-1c,us-east-1d

cloud:
aws:
access_key: xxxxxxxxxxxxxxxxx
secret_key: xxxxxxxxxxxxxxxxx
region: us-east-1
node:
auto_attributes: true

discovery:
type: ec2
ec2:
groups: elasticsearch
host_type: private_ip
availability_zones: us-east-1a,us-east-1c,us-
east-1d
any_group: false
ping_timeout: 5
tag:
stage: dev

On Dec 14, 5:54 pm, Shay Banon kim...@gmail.com wrote:

Yes, you will need to explicitly set the awareness in the routing
allocation forced part. You can set cloud.node.auto_attributes to true and
it will automatically add the ec2 relevant (node.aws_availability_zone) to
the node attributes.

On Wed, Dec 14, 2011 at 8:26 AM, Piavlo lolitus...@gmail.com wrote:

Hi,

I'm setting up Elasticsearch cluster in ec2

This is the discovery configuration

discovery:
type: ec2
ec2:
groups: elasticsearch
host_type: private_ip
availability_zones: us-east-1a,us-east-1c,us-
east-1d
any_group: false
ping_timeout: 5
tag:
stage: dev

Now

From the documentation it's not clear if Cluster module allocation
awareness makes use of the fact that node is running in ec2 - which
seems natural to do.

So the question is do I need to set routing.allocation.awareness
explicitly like this:

cluster:
name: dev
routing.allocation.awareness:
attributes:
availability_zone
force.availability_zone.values: us-east-1a,us-
east-1c,us-east-1d

node:
availability_zone: us-east-1a

Or this could be simplified or completely omitted with same effect?
Like specifying node.availability_zone - seems to be unneeded - as we
already know such info from the ec2 discovery module.

Thanks
Alex

4

It looks good, though it is hard to verify the structure of it because you
pasted it in a mail (gist is better).

One thing that I would verify is that the correct node attributes for the
availability zones are set, that should be simple to see with node info API.

On Wed, Dec 14, 2011 at 8:15 PM, Piavlo lolitushka@gmail.com wrote:

Hi Shay

Thanks for the rpley

To be sure I understood you correctly - could you please confirm that
the following is a full correct config to do the job?

cluster:
name: dev
routing.allocation.awareness:
attributes:
aws_availability_zone
force.aws_availability_zone.values: us-east-1a,us-
east-1c,us-east-1d

cloud:
aws:
access_key: xxxxxxxxxxxxxxxxx
secret_key: xxxxxxxxxxxxxxxxx
region: us-east-1
node:
auto_attributes: true

discovery:
type: ec2
ec2:
groups: elasticsearch
host_type: private_ip
availability_zones: us-east-1a,us-east-1c,us-
east-1d
any_group: false
ping_timeout: 5
tag:
stage: dev

On Dec 14, 5:54 pm, Shay Banon kim...@gmail.com wrote:

Yes, you will need to explicitly set the awareness in the routing
allocation forced part. You can set cloud.node.auto_attributes to true
and
it will automatically add the ec2 relevant (node.aws_availability_zone)
to
the node attributes.

On Wed, Dec 14, 2011 at 8:26 AM, Piavlo lolitus...@gmail.com wrote:

Hi,

I'm setting up Elasticsearch cluster in ec2

This is the discovery configuration

discovery:
type: ec2
ec2:
groups: elasticsearch
host_type: private_ip
availability_zones: us-east-1a,us-east-1c,us-
east-1d
any_group: false
ping_timeout: 5
tag:
stage: dev

Now

From the documentation it's not clear if Cluster module allocation
awareness makes use of the fact that node is running in ec2 - which
seems natural to do.

So the question is do I need to set routing.allocation.awareness
explicitly like this:

cluster:
name: dev
routing.allocation.awareness:
attributes:
availability_zone
force.availability_zone.values: us-east-1a,us-
east-1c,us-east-1d

node:
availability_zone: us-east-1a

Or this could be simplified or completely omitted with same effect?
Like specifying node.availability_zone - seems to be unneeded - as we
already know such info from the ec2 discovery module.

Thanks
Alex