Problem with cluster setup in AWS


(rohit reddy) #1

Hi,

I'm experimenting with elastic search on AWS. I'm following the tutorial "
http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html
".

I was able to successfully set up the elasticsearch instance on 1
Instance. But having problem setting it up of 2 instances. That is the 2
node is not able to discover the first node.
Both are up independently but replication is not happening.

I have used all the default configurations (5 shards and 1 replica) and
just changed the following properties. I using local storage, did not
configure s3 gateway

cluster.name: iv-search

cloud:
aws:
access_key:
secret_key:
discovery:
type: ec2
cloud.aws.region: us-east-1

I have also opened 22, 9300 and 9200 ports.
This is the my first time with both AWS and elasticsearch. So any help
would be great.

LOGS:

[2012-06-14 07:07:14,798][INFO ][node ] [Xavin]
{0.19.4}[2355]: started
[2012-06-14 07:07:14,810][INFO ][gateway ] [Xavin]
recovered [0] indices into cluster_state
[2012-06-14 09:40:18,845][INFO ][node ] [Zeus]
{0.19.4}[2515]: initializing ...
[2012-06-14 09:40:18,859][INFO ][plugins ] [Zeus] loaded
[cloud-aws], sites []
[2012-06-14 09:40:20,236][DEBUG][discovery.zen.ping.multicast] [Zeus] using
group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2012-06-14 09:40:20,239][DEBUG][discovery.zen.ping.unicast] [Zeus] using
initial hosts [], with concurrent_connects [10]
[2012-06-14 09:40:20,240][DEBUG][discovery.zen ] [Zeus] using
ping.timeout [3s]
[2012-06-14 09:40:20,245][DEBUG][discovery.zen.elect ] [Zeus] using
minimum_master_nodes [-1]
[2012-06-14 09:40:20,246][DEBUG][discovery.zen.fd ] [Zeus] [master]
uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-06-14 09:40:20,250][DEBUG][discovery.zen.fd ] [Zeus] [node ]
uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-06-14 09:40:21,298][DEBUG][gateway.local ] [Zeus] using
initial_shards [quorum], list_timeout [30s]
[2012-06-14 09:40:21,538][DEBUG][gateway.local.state.shards] [Zeus] took 0s
to load started shards state
[2012-06-14 09:40:21,540][DEBUG][gateway.local.state.meta ] [Zeus] took 0s
to load state
[2012-06-14 09:40:21,603][INFO ][node ] [Zeus]
{0.19.4}[2515]: initialized
[2012-06-14 09:40:21,604][INFO ][node ] [Zeus]
{0.19.4}[2515]: starting ...
[2012-06-14 09:40:21,685][INFO ][transport ] [Zeus]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.2.5.143:9300]}
[2012-06-14 09:40:21,697][TRACE][discovery ] [Zeus] waiting
for 30s for the initial state to be set by the discovery
[2012-06-14 09:40:21,725][TRACE][discovery.zen.ping.multicast] [Zeus] [1]
sending ping request
[2012-06-14 09:40:23,227][TRACE][discovery.zen.ping.multicast] [Zeus] [1]
sending ping request
[2012-06-14 09:40:24,729][DEBUG][discovery.zen ] [Zeus] ping
responses: {none}
[2012-06-14 09:40:24,734][INFO ][cluster.service ] [Zeus]
new_master [Zeus][secret][inet[/10.2.5.143:9300]], reason: zen-disco-join
(elected_as_master)
[2012-06-14 09:40:24,801][TRACE][discovery ] [Zeus] initial
state set from discovery
[2012-06-14 09:40:24,802][INFO ][discovery ] [Zeus]
iv-search/
[2012-06-14 09:40:24,816][INFO ][gateway ] [Zeus]
recovered [0] indices into cluster_state
[2012-06-14 09:40:24,818][INFO ][http ] [Zeus]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.2.5.143:9200]}
[2012-06-14 09:40:24,818][INFO ][node ] [Zeus]
{0.19.4}[2515]: started

Thanks
Rohit


(benny.sadeh) #2

your issue might be security-group related.
so first, just try allowing access to all, not just the security group on
ports 9200 & 9300.
if that solves the problem and you still want to restrict access, browse
the mailing list for related advice.

Benny

On Thursday, June 14, 2012 4:35:17 AM UTC-7, rohit reddy wrote:

Hi,

I'm experimenting with elastic search on AWS. I'm following the tutorial
"
http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html
".

I was able to successfully set up the elasticsearch instance on 1
Instance. But having problem setting it up of 2 instances. That is the 2
node is not able to discover the first node.
Both are up independently but replication is not happening.

I have used all the default configurations (5 shards and 1 replica) and
just changed the following properties. I using local storage, did not
configure s3 gateway

cluster.name: iv-search

cloud:
aws:
access_key:
secret_key:
discovery:
type: ec2
cloud.aws.region: us-east-1

I have also opened 22, 9300 and 9200 ports.
This is the my first time with both AWS and elasticsearch. So any help
would be great.

LOGS:

[2012-06-14 07:07:14,798][INFO ][node ] [Xavin]
{0.19.4}[2355]: started
[2012-06-14 07:07:14,810][INFO ][gateway ] [Xavin]
recovered [0] indices into cluster_state
[2012-06-14 09:40:18,845][INFO ][node ] [Zeus]
{0.19.4}[2515]: initializing ...
[2012-06-14 09:40:18,859][INFO ][plugins ] [Zeus] loaded
[cloud-aws], sites []
[2012-06-14 09:40:20,236][DEBUG][discovery.zen.ping.multicast] [Zeus]
using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2012-06-14 09:40:20,239][DEBUG][discovery.zen.ping.unicast] [Zeus] using
initial hosts [], with concurrent_connects [10]
[2012-06-14 09:40:20,240][DEBUG][discovery.zen ] [Zeus] using
ping.timeout [3s]
[2012-06-14 09:40:20,245][DEBUG][discovery.zen.elect ] [Zeus] using
minimum_master_nodes [-1]
[2012-06-14 09:40:20,246][DEBUG][discovery.zen.fd ] [Zeus]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-06-14 09:40:20,250][DEBUG][discovery.zen.fd ] [Zeus] [node
] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-06-14 09:40:21,298][DEBUG][gateway.local ] [Zeus] using
initial_shards [quorum], list_timeout [30s]
[2012-06-14 09:40:21,538][DEBUG][gateway.local.state.shards] [Zeus] took
0s to load started shards state
[2012-06-14 09:40:21,540][DEBUG][gateway.local.state.meta ] [Zeus] took 0s
to load state
[2012-06-14 09:40:21,603][INFO ][node ] [Zeus]
{0.19.4}[2515]: initialized
[2012-06-14 09:40:21,604][INFO ][node ] [Zeus]
{0.19.4}[2515]: starting ...
[2012-06-14 09:40:21,685][INFO ][transport ] [Zeus]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.2.5.143:9300]}
[2012-06-14 09:40:21,697][TRACE][discovery ] [Zeus] waiting
for 30s for the initial state to be set by the discovery
[2012-06-14 09:40:21,725][TRACE][discovery.zen.ping.multicast] [Zeus] [1]
sending ping request
[2012-06-14 09:40:23,227][TRACE][discovery.zen.ping.multicast] [Zeus] [1]
sending ping request
[2012-06-14 09:40:24,729][DEBUG][discovery.zen ] [Zeus] ping
responses: {none}
[2012-06-14 09:40:24,734][INFO ][cluster.service ] [Zeus]
new_master [Zeus][secret][inet[/10.2.5.143:9300]], reason: zen-disco-join
(elected_as_master)
[2012-06-14 09:40:24,801][TRACE][discovery ] [Zeus] initial
state set from discovery
[2012-06-14 09:40:24,802][INFO ][discovery ] [Zeus]
iv-search/
[2012-06-14 09:40:24,816][INFO ][gateway ] [Zeus]
recovered [0] indices into cluster_state
[2012-06-14 09:40:24,818][INFO ][http ] [Zeus]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.2.5.143:9200]}
[2012-06-14 09:40:24,818][INFO ][node ] [Zeus]
{0.19.4}[2515]: started

Thanks
Rohit


(rohit reddy) #3

Thanks for the reply.
The setup started working when i used the versions mentioned in the
tutorial. i.e.,
elasticsearch : 0.19.0
elasticsearch-cloud-aws : 1.4.0

It was not working with elasticsearch 0.19.4 and cloud-aws 1.6.0. Did the
configurations change in these versions.
I search the user mail. Did not get anything relevant

Thanks
Rohit

On Tuesday, June 19, 2012 7:20:27 PM UTC+5:30, benny.sadeh wrote:

your issue might be security-group related.
so first, just try allowing access to all, not just the security group on
ports 9200 & 9300.
if that solves the problem and you still want to restrict access, browse
the mailing list for related advice.

Benny

On Thursday, June 14, 2012 4:35:17 AM UTC-7, rohit reddy wrote:

Hi,

I'm experimenting with elastic search on AWS. I'm following the
tutorial "
http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html
".

I was able to successfully set up the elasticsearch instance on 1
Instance. But having problem setting it up of 2 instances. That is the 2
node is not able to discover the first node.
Both are up independently but replication is not happening.

I have used all the default configurations (5 shards and 1 replica) and
just changed the following properties. I using local storage, did not
configure s3 gateway

cluster.name: iv-search

cloud:
aws:
access_key:
secret_key:
discovery:
type: ec2
cloud.aws.region: us-east-1

I have also opened 22, 9300 and 9200 ports.
This is the my first time with both AWS and elasticsearch. So any help
would be great.

LOGS:

[2012-06-14 07:07:14,798][INFO ][node ] [Xavin]
{0.19.4}[2355]: started
[2012-06-14 07:07:14,810][INFO ][gateway ] [Xavin]
recovered [0] indices into cluster_state
[2012-06-14 09:40:18,845][INFO ][node ] [Zeus]
{0.19.4}[2515]: initializing ...
[2012-06-14 09:40:18,859][INFO ][plugins ] [Zeus] loaded
[cloud-aws], sites []
[2012-06-14 09:40:20,236][DEBUG][discovery.zen.ping.multicast] [Zeus]
using group [224.2.2.4], with port [54328], ttl [3], and address [null]
[2012-06-14 09:40:20,239][DEBUG][discovery.zen.ping.unicast] [Zeus] using
initial hosts [], with concurrent_connects [10]
[2012-06-14 09:40:20,240][DEBUG][discovery.zen ] [Zeus] using
ping.timeout [3s]
[2012-06-14 09:40:20,245][DEBUG][discovery.zen.elect ] [Zeus] using
minimum_master_nodes [-1]
[2012-06-14 09:40:20,246][DEBUG][discovery.zen.fd ] [Zeus]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-06-14 09:40:20,250][DEBUG][discovery.zen.fd ] [Zeus] [node
] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2012-06-14 09:40:21,298][DEBUG][gateway.local ] [Zeus] using
initial_shards [quorum], list_timeout [30s]
[2012-06-14 09:40:21,538][DEBUG][gateway.local.state.shards] [Zeus] took
0s to load started shards state
[2012-06-14 09:40:21,540][DEBUG][gateway.local.state.meta ] [Zeus] took
0s to load state
[2012-06-14 09:40:21,603][INFO ][node ] [Zeus]
{0.19.4}[2515]: initialized
[2012-06-14 09:40:21,604][INFO ][node ] [Zeus]
{0.19.4}[2515]: starting ...
[2012-06-14 09:40:21,685][INFO ][transport ] [Zeus]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.2.5.143:9300]}
[2012-06-14 09:40:21,697][TRACE][discovery ] [Zeus]
waiting for 30s for the initial state to be set by the discovery
[2012-06-14 09:40:21,725][TRACE][discovery.zen.ping.multicast] [Zeus] [1]
sending ping request
[2012-06-14 09:40:23,227][TRACE][discovery.zen.ping.multicast] [Zeus] [1]
sending ping request
[2012-06-14 09:40:24,729][DEBUG][discovery.zen ] [Zeus] ping
responses: {none}
[2012-06-14 09:40:24,734][INFO ][cluster.service ] [Zeus]
new_master [Zeus][secret][inet[/10.2.5.143:9300]], reason: zen-disco-join
(elected_as_master)
[2012-06-14 09:40:24,801][TRACE][discovery ] [Zeus]
initial state set from discovery
[2012-06-14 09:40:24,802][INFO ][discovery ] [Zeus]
iv-search/
[2012-06-14 09:40:24,816][INFO ][gateway ] [Zeus]
recovered [0] indices into cluster_state
[2012-06-14 09:40:24,818][INFO ][http ] [Zeus]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.2.5.143:9200]}
[2012-06-14 09:40:24,818][INFO ][node ] [Zeus]
{0.19.4}[2515]: started

Thanks
Rohit


(system) #4