Hi,
I'm trying to configure two Elasticsearch nodes in AWS in the same
autoscaling group (CloudFormation).
I am having some problems with them discovering each other.
The following shows the elasticsearch.log I have on the first machine with
the instance-id "i-2db5db03".
The second machine has an instance-id "i-324e6612".
It seems that both nodes recognize each other, thanks to
discovery.ec2.tag.* field I added but then there are some problems that
make them not to join together:
[2014-02-07 13:17:08,852][INFO ][node ]
[ip-10-238-225-133.ec2.internal] version[1.0.0.Beta2], pid[15342],
build[296cfbe/2013-12-02T15:46:27Z]
[2014-02-07 13:17:08,853][INFO ][node ]
[ip-10-238-225-133.ec2.internal] initializing ...
[2014-02-07 13:17:08,917][INFO ][plugins ]
[ip-10-238-225-133.ec2.internal] loaded [cloud-aws], sites [paramedic]
[2014-02-07 13:17:15,452][DEBUG][discovery.zen.ping.unicast]
[ip-10-238-225-133.ec2.internal] using initial hosts [], with
concurrent_connects [10]
[2014-02-07 13:17:15,455][DEBUG][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] using ping.timeout [3s],
master_election.filter_client [true], master_election.filter_data [false]
[2014-02-07 13:17:15,456][DEBUG][discovery.zen.elect ]
[ip-10-238-225-133.ec2.internal] using minimum_master_nodes [1]
[2014-02-07 13:17:15,457][DEBUG][discovery.zen.fd ]
[ip-10-238-225-133.ec2.internal] [master] uses ping_interval [1s],
ping_timeout [30s], ping_retries [3]
[2014-02-07 13:17:15,500][DEBUG][discovery.zen.fd ]
[ip-10-238-225-133.ec2.internal] [node ] uses ping_interval [1s],
ping_timeout [30s], ping_retries [3]
[2014-02-07 13:17:16,769][DEBUG][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] using host_type [PRIVATE_IP], tags
[{platform=prod}], groups [[]] with any_group [true], availability_zones
[[]]
[2014-02-07 13:17:19,930][INFO ][node ]
[ip-10-238-225-133.ec2.internal] initialized
[2014-02-07 13:17:19,931][INFO ][node ]
[ip-10-238-225-133.ec2.internal] starting ...
[2014-02-07 13:17:20,455][INFO ][transport ]
[ip-10-238-225-133.ec2.internal] bound_address {inet[/0.0.0.0:9300]},
publish_address {inet[/10.238.225.133:9300]}
[2014-02-07 13:17:20,527][TRACE][discovery ]
[ip-10-238-225-133.ec2.internal] waiting for 30s for the initial state to
be set by the discovery
[2014-02-07 13:17:21,981][TRACE][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] building dynamic unicast discovery nodes...
[2014-02-07 13:17:21,982][TRACE][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] filtering out instance i-2db5db03 based
tags {platform=prod}, not part of [{Key: aws:cloudformation:stack-id,
Value:
arn:aws:cloudformation:us-east-1:876119091332:stack/ES-10/daf53050-8ff8-11e3-bdce-50e241629418,
}, {Key: aws:cloudformation:stack-name, Value: ES-10, }, {Key:
aws:cloudformation:logical-id, Value: ESASG, }, {Key:
aws:autoscaling:groupName, Value: ES-10-ESASG-BHGX7KKQ9QPR, }]
[2014-02-07 13:17:21,983][TRACE][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] filtering out instance i-324e6612 based
tags {platform=prod}, not part of [{Key: aws:cloudformation:logical-id,
Value: ESASG, }, {Key: aws:cloudformation:stack-id, Value:
arn:aws:cloudformation:us-east-1:876119091332:stack/ES-10/daf53050-8ff8-11e3-bdce-50e241629418,
}, {Key: aws:cloudformation:stack-name, Value: ES-10, }, {Key:
aws:autoscaling:groupName, Value: ES-10-ESASG-BHGX7KKQ9QPR, }]
[2014-02-07 13:17:21,983][DEBUG][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] using dynamic discovery nodes []
[2014-02-07 13:17:23,744][TRACE][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] building dynamic unicast discovery nodes...
[2014-02-07 13:17:23,745][TRACE][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] filtering out instance i-2db5db03 based
tags {platform=prod}, not part of [{Key: aws:cloudformation:stack-id,
Value:
arn:aws:cloudformation:us-east-1:876119091332:stack/ES-10/daf53050-8ff8-11e3-bdce-50e241629418,
}, {Key: aws:cloudformation:stack-name, Value: ES-10, }, {Key:
aws:cloudformation:logical-id, Value: ESASG, }, {Key:
aws:autoscaling:groupName, Value: ES-10-ESASG-BHGX7KKQ9QPR, }]
[2014-02-07 13:17:23,745][TRACE][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] filtering out instance i-324e6612 based
tags {platform=prod}, not part of [{Key: aws:cloudformation:logical-id,
Value: ESASG, }, {Key: aws:cloudformation:stack-id, Value:
arn:aws:cloudformation:us-east-1:876119091332:stack/ES-10/daf53050-8ff8-11e3-bdce-50e241629418,
}, {Key: aws:cloudformation:stack-name, Value: ES-10, }, {Key:
aws:autoscaling:groupName, Value: ES-10-ESASG-BHGX7KKQ9QPR, }]
[2014-02-07 13:17:25,604][DEBUG][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] using dynamic discovery nodes []
[2014-02-07 13:17:25,604][TRACE][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] full ping responses: {none}
[2014-02-07 13:17:25,605][DEBUG][discovery.ec2 ]
[ip-10-238-225-133.ec2.internal] filtered ping responses:
(filter_client[true], filter_data[false]) {none}
[2014-02-07 13:17:25,618][INFO ][cluster.service ]
[ip-10-238-225-133.ec2.internal] new_master
[ip-10-238-225-133.ec2.internal][tUBe7JNdTquROI6UWVLNpw][inet[/10.238.225.133:9300]]{aws_availability_zone=us-east-1d,
max_local_storage_nodes=1}, reason: zen-disco-join (elected_as_master)
[2014-02-07 13:17:25,752][TRACE][discovery ]
[ip-10-238-225-133.ec2.internal] initial state set from discovery
[2014-02-07 13:17:25,752][INFO ][discovery ]
[ip-10-238-225-133.ec2.internal] elasticsearch/tUBe7JNdTquROI6UWVLNpw
[2014-02-07 13:17:25,892][INFO ][http ]
[ip-10-238-225-133.ec2.internal] bound_address {inet[/0.0.0.0:9200]},
publish_address {inet[/10.238.225.133:9200]}
[2014-02-07 13:17:25,893][INFO ][node ]
[ip-10-238-225-133.ec2.internal] started
[2014-02-07 13:17:25,924][INFO ][gateway ]
[ip-10-238-225-133.ec2.internal] recovered [0] indices into cluster_state
I'm using Chef to configure my node with the cookbook of Karmi
(https://github.com/elasticsearch/cookbook-elasticsearch).
The node.json is the following:
"/etc/chef/node.json": {
"content": {
"run_list": [
"recipe[java]",
"recipe[monit]",
"recipe[elasticsearch]",
"recipe[elasticsearch::plugins]",
"recipe[elasticsearch::aws]",
"recipe[elasticsearch::monit]"
],
"java": {
"install_flavor": "oracle",
"oracle": {
"accept_oracle_download_terms": true
},
"jdk_version": "7"
},
"elasticsearch": {
"version": "1.0.0.Beta2",
"logging": {
"discovery": "TRACE"
},
"cluster": {
"name": "elasticsearch"
},
"bootstrap": {
"mlockall": false
},
"discovery": {
"type": "ec2",
"ec2": {
"groups": {"Ref": "ESSecurityGroup"},
"tag": {
"platform": "prod"
}
},
"zen": {
"ping": {
"multicast": {
"enabled": false
}
}
}
},
"cloud": {
"aws": {
"access_key": {"Ref": "ESHostKeys"},
"secret_key": {"Fn::GetAtt": [
"ESHostKeys", "SecretAccessKey"]},
"region": "us-east-1"
},
"ec2": {
"security_group": {"Ref": "ESSecurityGroup"
}
}
},
"monit": {
"notify_email": "my_email_address",
"mail_format": {
"from": "from_email_address",
"subject": "[monit]$SERVICE$EVENTon$HOST"
,
"message": "$SERVICE$ACTION:
$DESCRIPTION"
}
},
"plugins": {
"karmi/elasticsearch-paramedic" : {}
}
}
},
"mode": "000644",
"owner": "root",
"group": "root"
}
}
I'm using Oracle 7 for ElasticSearch. The machines are Ubuntu 12.04 and I'm
using Elasticsearch 1.0.0 RC2.
It seems that the first node is seeing the second node but refuse to join
it because the tags "aws:cloudformation:stack-id", "
aws:cloudformation:stack-name", "aws:cloudformation:logical-id" and "aws:autoscaling:groupName"
are not the same. However, I checked these tags for both instances and it's
the same (as the two instances have been created in the same Autoscaling
group).
Any ideas?
Thanks!
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6247ec9a-e7b5-4dba-8f8d-a38fde5704bc%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.