AWS nodes don't discover each other

gzion7 · October 30, 2013, 8:13pm

I have two ES servers in AWS, I have security groups set to allow all
traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ] [Burstarr]
version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:50:20Z]
[2013-10-30 16:11:12,703][INFO ][node ] [Burstarr]
initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ] [Burstarr]
loaded [cloud-aws], sites []
[2013-10-30 16:11:15,199][DEBUG][discovery.zen.ping.unicast] [Burstarr]
using initial hosts [], with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][discovery.zen ] [Burstarr]
using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][discovery.zen.elect ] [Burstarr]
using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][discovery.zen.fd ] [Burstarr]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][discovery.zen.fd ] [Burstarr]
[node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.local ] [Burstarr]
using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.recovery ] [Burstarr]
using max_bytes_per_sec[20mb], concurrent_streams [3], file_chunk_size
[512kb], translog_size [512kb], translog_ops [1000], and compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.local.state.meta ] [Burstarr]
using gateway.local.auto_import_dangled [YES], with
gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.meta ] [Burstarr] took
25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.shards] [Burstarr]
took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ] [Burstarr]
initialized
[2013-10-30 16:11:16,747][INFO ][node ] [Burstarr]
starting ...
[2013-10-30 16:11:16,882][INFO ][transport ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][discovery ] [Burstarr]
waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][discovery.zen ] [Burstarr] full
ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][discovery.zen ] [Burstarr]
filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ] [Burstarr]
new_master [Burstarr][pRsc595IQfyXn24N27tbZw][inet[/10.20.4.158:9300]],
reason: zen-disco-join (elected_as_master)
[2013-10-30 16:11:19,977][TRACE][discovery ] [Burstarr]
initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ] [Burstarr]
elasticsearch/pRsc595IQfyXn24N27tbZw
[2013-10-30 16:11:20,012][INFO ][http ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ] [Burstarr]
started
[2013-10-30 16:11:20,025][INFO ][gateway ] [Burstarr]
recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Norberto_Meijome · October 31, 2013, 1:42am

It seems you didn't set discovery type ec2. And obviously u need to provide
API keys...
And you should also tell it which filter to use ( ie by tag or security
group..otherwise it will attempt to connect to any of your hosts...which
may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" gavysdomain@gmail.com wrote:

I have two ES servers in AWS, I have security groups set to allow all
traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ] [Burstarr]
version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:50:20Z]
[2013-10-30 16:11:12,703][INFO ][node ] [Burstarr]
initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ] [Burstarr]
loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][discovery.zen.ping.unicast] [Burstarr]
using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][discovery.zen ] [Burstarr]
using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][discovery.zen.elect ] [Burstarr]
using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][discovery.zen.fd ] [Burstarr]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][discovery.zen.fd ] [Burstarr]
[node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.local ] [Burstarr]
using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.recovery ] [Burstarr]
using max_bytes_per_sec[20mb], concurrent_streams [3], file_chunk_size
[512kb], translog_size [512kb], translog_ops [1000], and compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.local.state.meta ] [Burstarr]
using gateway.local.auto_import_dangled [YES], with
gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.meta ] [Burstarr]
took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.shards] [Burstarr]
took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ] [Burstarr]
initialized
[2013-10-30 16:11:16,747][INFO ][node ] [Burstarr]
starting ...
[2013-10-30 16:11:16,882][INFO ][transport ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][discovery ] [Burstarr]
waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][discovery.zen ] [Burstarr]
full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][discovery.zen ] [Burstarr]
filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ] [Burstarr]
new_master [Burstarr][pRsc595IQfyXn24N27tbZw][inet[/10.20.4.158:9300]],
reason: zen-disco-join (elected_as_master)
[2013-10-30 16:11:19,977][TRACE][discovery ] [Burstarr]
initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ] [Burstarr]
elasticsearch/pRsc595IQfyXn24N27tbZw
[2013-10-30 16:11:20,012][INFO ][http ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ] [Burstarr]
started
[2013-10-30 16:11:20,025][INFO ][gateway ] [Burstarr]
recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

gzion7 · October 31, 2013, 1:51am

You mean on the bottom of the file where I add my keys? I added those
already
How can I tag by security groups?

On Wednesday, October 30, 2013 9:42:23 PM UTC-4, Norberto Meijome wrote:

It seems you didn't set discovery type ec2. And obviously u need to
provide API keys...
And you should also tell it which filter to use ( ie by tag or security
group..otherwise it will attempt to connect to any of your hosts...which
may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" <gavys...@gmail.com <javascript:>>
wrote:

I have two ES servers in AWS, I have security groups set to allow all
traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ] [Burstarr]
version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:50:20Z]
[2013-10-30 16:11:12,703][INFO ][node ] [Burstarr]
initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ] [Burstarr]
loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][discovery.zen.ping.unicast] [Burstarr]
using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][discovery.zen ] [Burstarr]
using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][discovery.zen.elect ] [Burstarr]
using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][discovery.zen.fd ] [Burstarr]
[master] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][discovery.zen.fd ] [Burstarr]
[node ] uses ping_interval [1s], ping_timeout [30s], ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.local ] [Burstarr]
using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.recovery ] [Burstarr]
using max_bytes_per_sec[20mb], concurrent_streams [3], file_chunk_size
[512kb], translog_size [512kb], translog_ops [1000], and compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.local.state.meta ] [Burstarr]
using gateway.local.auto_import_dangled [YES], with
gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.meta ] [Burstarr]
took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.shards] [Burstarr]
took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ] [Burstarr]
initialized
[2013-10-30 16:11:16,747][INFO ][node ] [Burstarr]
starting ...
[2013-10-30 16:11:16,882][INFO ][transport ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][discovery ] [Burstarr]
waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][discovery.zen ] [Burstarr]
full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][discovery.zen ] [Burstarr]
filtered ping responses: (filter_client[true], filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ] [Burstarr]
new_master [Burstarr][pRsc595IQfyXn24N27tbZw][inet[/10.20.4.158:9300]],
reason: zen-disco-join (elected_as_master)
[2013-10-30 16:11:19,977][TRACE][discovery ] [Burstarr]
initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ] [Burstarr]
elasticsearch/pRsc595IQfyXn24N27tbZw
[2013-10-30 16:11:20,012][INFO ][http ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ] [Burstarr]
started
[2013-10-30 16:11:20,025][INFO ][gateway ] [Burstarr]
recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Norberto_Meijome · October 31, 2013, 7:36am

It's all in the docs for the ec2 plugin.
On 31/10/2013 12:51 PM, "Gabriel Holder" gavysdomain@gmail.com wrote:

You mean on the bottom of the file where I add my keys? I added those
already
How can I tag by security groups?

On Wednesday, October 30, 2013 9:42:23 PM UTC-4, Norberto Meijome wrote:

It seems you didn't set discovery type ec2. And obviously u need to
provide API keys...
And you should also tell it which filter to use ( ie by tag or security
group..otherwise it will attempt to connect to any of your hosts...which
may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" gavys...@gmail.com wrote:

I have two ES servers in AWS, I have security groups set to allow all
traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ] [Burstarr]
version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:**50:20Z]
[2013-10-30 16:11:12,703][INFO ][node ] [Burstarr]
initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ] [Burstarr]
loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][**discovery.zen.ping.unicast]
[Burstarr] using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][**discovery.zen ]
[Burstarr] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][**discovery.zen.elect ]
[Burstarr] using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][**discovery.zen.fd ]
[Burstarr] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][**discovery.zen.fd ]
[Burstarr] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.**local ]
[Burstarr] using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.**recovery ]
[Burstarr] using max_bytes_per_sec[20mb], concurrent_streams [3],
file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and
compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.**local.state.meta ]
[Burstarr] using gateway.local.auto_import_**dangled [YES], with
gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.**local.state.meta ]
[Burstarr] took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.**local.state.shards]
[Burstarr] took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ] [Burstarr]
initialized
[2013-10-30 16:11:16,747][INFO ][node ] [Burstarr]
starting ...
[2013-10-30 16:11:16,882][INFO ][transport ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][**discovery ]
[Burstarr] waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][**discovery.zen ]
[Burstarr] full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][**discovery.zen ]
[Burstarr] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ] [Burstarr]
new_master [Burstarr][**pRsc595IQfyXn24N27tbZw][inet[/**10.20.4.158:9300]],
reason: zen-disco-join (elected_as_master)
[2013-10-30 16:11:19,977][TRACE][**discovery ]
[Burstarr] initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ] [Burstarr]
elasticsearch/**pRsc595IQfyXn24N27tbZw
[2013-10-30 16:11:20,012][INFO ][http ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ] [Burstarr]
started
[2013-10-30 16:11:20,025][INFO ][gateway ] [Burstarr]
recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

gzion7 · October 31, 2013, 2:54pm

This is what I've added:
cloud:
aws:
access_key:
secret_key:

discovery:
type: ec2
groups: SECURITY-GROUP

Unfortunately, they still don't see each other

On Thursday, October 31, 2013 3:36:56 AM UTC-4, Norberto Meijome wrote:

It's all in the docs for the ec2 plugin.
On 31/10/2013 12:51 PM, "Gabriel Holder" <gavys...@gmail.com <javascript:>>
wrote:

You mean on the bottom of the file where I add my keys? I added those
already
How can I tag by security groups?

On Wednesday, October 30, 2013 9:42:23 PM UTC-4, Norberto Meijome wrote:

It seems you didn't set discovery type ec2. And obviously u need to
provide API keys...
And you should also tell it which filter to use ( ie by tag or security
group..otherwise it will attempt to connect to any of your hosts...which
may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" gavys...@gmail.com wrote:

I have two ES servers in AWS, I have security groups set to allow all
traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ] [Burstarr]
version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:**50:20Z]
[2013-10-30 16:11:12,703][INFO ][node ] [Burstarr]
initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ] [Burstarr]
loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][**discovery.zen.ping.unicast]
[Burstarr] using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][**discovery.zen ]
[Burstarr] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][**discovery.zen.elect ]
[Burstarr] using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][**discovery.zen.fd ]
[Burstarr] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][**discovery.zen.fd ]
[Burstarr] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.**local ]
[Burstarr] using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.**recovery ]
[Burstarr] using max_bytes_per_sec[20mb], concurrent_streams [3],
file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and
compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.**local.state.meta ]
[Burstarr] using gateway.local.auto_import_**dangled [YES], with
gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.**local.state.meta ]
[Burstarr] took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.**local.state.shards]
[Burstarr] took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ] [Burstarr]
initialized
[2013-10-30 16:11:16,747][INFO ][node ] [Burstarr]
starting ...
[2013-10-30 16:11:16,882][INFO ][transport ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][**discovery ]
[Burstarr] waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][**discovery.zen ]
[Burstarr] full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][**discovery.zen ]
[Burstarr] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ] [Burstarr]
new_master [Burstarr][**pRsc595IQfyXn24N27tbZw][inet[/**10.20.4.158:9300]],
reason: zen-disco-join (elected_as_master)
[2013-10-30 16:11:19,977][TRACE][**discovery ]
[Burstarr] initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ] [Burstarr]
elasticsearch/**pRsc595IQfyXn24N27tbZw
[2013-10-30 16:11:20,012][INFO ][http ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ] [Burstarr]
started
[2013-10-30 16:11:20,025][INFO ][gateway ] [Burstarr]
recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

gzion7 · October 31, 2013, 4:37pm

Update:
I managed to get it to work. I did not realize how finicky the YML file is
with the correct spacing.
I needed to add an EIP so it could check my AWS credentials on one of their
sites.

Can I create a NAT instance to fix this? I don't want these servers having
a public IP.

On Thursday, October 31, 2013 10:54:34 AM UTC-4, Gabriel Holder wrote:

This is what I've added:
cloud:
aws:
access_key:
secret_key:
discovery:
type: ec2
groups: SECURITY-GROUP
Unfortunately, they still don't see each other

On Thursday, October 31, 2013 3:36:56 AM UTC-4, Norberto Meijome wrote:

It's all in the docs for the ec2 plugin.
On 31/10/2013 12:51 PM, "Gabriel Holder" gavys...@gmail.com wrote:

You mean on the bottom of the file where I add my keys? I added those
already
How can I tag by security groups?

On Wednesday, October 30, 2013 9:42:23 PM UTC-4, Norberto Meijome wrote:

It seems you didn't set discovery type ec2. And obviously u need to
provide API keys...
And you should also tell it which filter to use ( ie by tag or security
group..otherwise it will attempt to connect to any of your hosts...which
may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" gavys...@gmail.com wrote:

I have two ES servers in AWS, I have security groups set to allow all
traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ] [Burstarr]
version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:**50:20Z]
[2013-10-30 16:11:12,703][INFO ][node ] [Burstarr]
initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ] [Burstarr]
loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][**discovery.zen.ping.unicast]
[Burstarr] using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][**discovery.zen ]
[Burstarr] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][**discovery.zen.elect ]
[Burstarr] using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][**discovery.zen.fd ]
[Burstarr] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][**discovery.zen.fd ]
[Burstarr] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.**local ]
[Burstarr] using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.**recovery ]
[Burstarr] using max_bytes_per_sec[20mb], concurrent_streams [3],
file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and
compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.**local.state.meta ]
[Burstarr] using gateway.local.auto_import_**dangled [YES], with
gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.**local.state.meta ]
[Burstarr] took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.**local.state.shards]
[Burstarr] took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ] [Burstarr]
initialized
[2013-10-30 16:11:16,747][INFO ][node ] [Burstarr]
starting ...
[2013-10-30 16:11:16,882][INFO ][transport ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][**discovery ]
[Burstarr] waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][**discovery.zen ]
[Burstarr] full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][**discovery.zen ]
[Burstarr] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ] [Burstarr]
new_master [Burstarr][**pRsc595IQfyXn24N27tbZw][inet[/**10.20.4.158:9300]],
reason: zen-disco-join (elected_as_master)
[2013-10-30 16:11:19,977][TRACE][**discovery ]
[Burstarr] initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ] [Burstarr]
elasticsearch/**pRsc595IQfyXn24N27tbZw
[2013-10-30 16:11:20,012][INFO ][http ] [Burstarr]
bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ] [Burstarr]
started
[2013-10-30 16:11:20,025][INFO ][gateway ] [Burstarr]
recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Norberto_Meijome · October 31, 2013, 10:22pm

I don't see why you would need a public IP... Just make sure:

your instances can reach each other via the protocol used for clustering
(TCP/9300 by default)
point above includes instances which may appear external to your
cluster.. Across regions?..
you API key can properly list instances (and describe tags in them if you
do that )
On 01/11/2013 3:37 AM, "Gabriel Holder" gavysdomain@gmail.com wrote:

Update:
I managed to get it to work. I did not realize how finicky the YML file is
with the correct spacing.
I needed to add an EIP so it could check my AWS credentials on one of
their sites.

Can I create a NAT instance to fix this? I don't want these servers having
a public IP.

On Thursday, October 31, 2013 10:54:34 AM UTC-4, Gabriel Holder wrote:
This is what I've added:
cloud:
aws:
access_key:
secret_key:
discovery:
type: ec2
groups: SECURITY-GROUP
Unfortunately, they still don't see each other

On Thursday, October 31, 2013 3:36:56 AM UTC-4, Norberto Meijome wrote:

It's all in the docs for the ec2 plugin.
On 31/10/2013 12:51 PM, "Gabriel Holder" gavys...@gmail.com wrote:

You mean on the bottom of the file where I add my keys? I added those
already
How can I tag by security groups?

On Wednesday, October 30, 2013 9:42:23 PM UTC-4, Norberto Meijome wrote:

It seems you didn't set discovery type ec2. And obviously u need to
provide API keys...
And you should also tell it which filter to use ( ie by tag or
security group..otherwise it will attempt to connect to any of your
hosts...which may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" gavys...@gmail.com wrote:

I have two ES servers in AWS, I have security groups set to allow all
traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ]
[Burstarr] version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:*
50*:20Z]
[2013-10-30 16:11:12,703][INFO ][node ]
[Burstarr] initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ]
[Burstarr] loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][discovery.zen.ping.unicast]
[Burstarr] using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][discovery.zen ]
[Burstarr] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][discovery.zen.elect ]
[Burstarr] using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][discovery.zen.fd ]
[Burstarr] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][discovery.zen.fd ]
[Burstarr] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.local ]
[Burstarr] using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.recovery ]
[Burstarr] using max_bytes_per_sec[20mb], concurrent_streams [3],
file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and
compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.local.state.meta ]
[Burstarr] using gateway.local.auto_import_dangled [YES], with
gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.meta ]
[Burstarr] took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.shards]
[Burstarr] took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ]
[Burstarr] initialized
[2013-10-30 16:11:16,747][INFO ][node ]
[Burstarr] starting ...
[2013-10-30 16:11:16,882][INFO ][transport ]
[Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][discovery ]
[Burstarr] waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][discovery.zen ]
[Burstarr] full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][discovery.zen ]
[Burstarr] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ]
[Burstarr] new_master [Burstarr][pRsc595IQfyXn24N27tbZw][inet[/**
10.20.4.158:9300]], reason: zen-disco-join (elected_as_master)
[2013-10-30 16:11:19,977][TRACE][discovery ]
[Burstarr] initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ]
[Burstarr] elasticsearch/pRsc595IQfyXn24N27tbZw
[2013-10-30 16:11:20,012][INFO ][http ]
[Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ]
[Burstarr] started
[2013-10-30 16:11:20,025][INFO ][gateway ]
[Burstarr] recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@**googlegroups.**com.
For more options, visit https://groups.google.com/**grou**ps/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

gzion7 · October 31, 2013, 10:39pm

I can telnet just fine to 9200, I'll have to test 9300.
These are within the same region.
Do you mean running ec2-describe-instances from the command line?

I think the reason why I need a public IP is because it tries to validate
my AWS credentials on their site. This could just be because at this moment
I don't have a
NAT instance setup so I had to actually add the discovery: ec2...etc part.

In my previous setup I didn't need to specify discovery:ec2 part in my
config with the NAT instance. They were able to see each other simply
because of the AWS Plugin.

On Thursday, October 31, 2013 6:22:29 PM UTC-4, Norberto Meijome wrote:

I don't see why you would need a public IP... Just make sure:

your instances can reach each other via the protocol used for clustering
(TCP/9300 by default)

point above includes instances which may appear external to your
cluster.. Across regions?..

you API key can properly list instances (and describe tags in them if
you do that )
On 01/11/2013 3:37 AM, "Gabriel Holder" <gavys...@gmail.com <javascript:>>
wrote:
Update:
I managed to get it to work. I did not realize how finicky the YML file
is with the correct spacing.
I needed to add an EIP so it could check my AWS credentials on one of
their sites.

Can I create a NAT instance to fix this? I don't want these servers
having a public IP.

On Thursday, October 31, 2013 10:54:34 AM UTC-4, Gabriel Holder wrote:
This is what I've added:
cloud:
aws:
access_key:
secret_key:
discovery:
type: ec2
groups: SECURITY-GROUP
Unfortunately, they still don't see each other

On Thursday, October 31, 2013 3:36:56 AM UTC-4, Norberto Meijome wrote:

It's all in the docs for the ec2 plugin.
On 31/10/2013 12:51 PM, "Gabriel Holder" gavys...@gmail.com wrote:

You mean on the bottom of the file where I add my keys? I added those
already
How can I tag by security groups?

On Wednesday, October 30, 2013 9:42:23 PM UTC-4, Norberto Meijome
wrote:

It seems you didn't set discovery type ec2. And obviously u need to
provide API keys...
And you should also tell it which filter to use ( ie by tag or
security group..otherwise it will attempt to connect to any of your
hosts...which may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" gavys...@gmail.com wrote:

I have two ES servers in AWS, I have security groups set to allow
all traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ]
[Burstarr] version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:
50:20Z]
[2013-10-30 16:11:12,703][INFO ][node ]
[Burstarr] initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ]
[Burstarr] loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][discovery.zen.ping.unicast]
[Burstarr] using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][discovery.zen ]
[Burstarr] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][discovery.zen.elect ]
[Burstarr] using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][discovery.zen.fd ]
[Burstarr] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][discovery.zen.fd ]
[Burstarr] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.local ]
[Burstarr] using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.recovery ]
[Burstarr] using max_bytes_per_sec[20mb], concurrent_streams [3],
file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and
compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.local.state.meta ]
[Burstarr] using gateway.local.auto_import_dangled [YES], with
gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.meta ]
[Burstarr] took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.shards]
[Burstarr] took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ]
[Burstarr] initialized
[2013-10-30 16:11:16,747][INFO ][node ]
[Burstarr] starting ...
[2013-10-30 16:11:16,882][INFO ][transport ]
[Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][discovery ]
[Burstarr] waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][discovery.zen ]
[Burstarr] full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][discovery.zen ]
[Burstarr] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ]
[Burstarr] new_master [Burstarr][pRsc595IQfyXn24N27tbZw][inet[/*
*10.20.4.158:9300]], reason: zen-disco-join (elected_as_master)
[2013-10-30 16:11:19,977][TRACE][discovery ]
[Burstarr] initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ]
[Burstarr] elasticsearch/pRsc595IQfyXn24N27tbZw
[2013-10-30 16:11:20,012][INFO ][http ]
[Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ]
[Burstarr] started
[2013-10-30 16:11:20,025][INFO ][gateway ]
[Burstarr] recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@**googlegroups.**com.
For more options, visit https://groups.google.com/**grou**ps/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.**com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Norberto_Meijome · November 1, 2013, 2:15am

9200 is http API port. Clustering goes over 9300 ( by default)
I mean the API key you use has to be able to get the information it needs
to find servers to speak to. It does this by connecting to AWS API and
listing instances filtered by what you tell it in the config. But yes, you
should be able to get this information manually from the CLI , as a
debugging step.
On 01/11/2013 9:39 AM, "Gabriel Holder" gavysdomain@gmail.com wrote:

I can telnet just fine to 9200, I'll have to test 9300.

These are within the same region.

Do you mean running ec2-describe-instances from the command line?

I think the reason why I need a public IP is because it tries to validate
my AWS credentials on their site. This could just be because at this moment
I don't have a
NAT instance setup so I had to actually add the discovery: ec2...etc part.

In my previous setup I didn't need to specify discovery:ec2 part in my
config with the NAT instance. They were able to see each other simply
because of the AWS Plugin.

On Thursday, October 31, 2013 6:22:29 PM UTC-4, Norberto Meijome wrote:
I don't see why you would need a public IP... Just make sure:

your instances can reach each other via the protocol used for
clustering (TCP/9300 by default)

point above includes instances which may appear external to your
cluster.. Across regions?..

you API key can properly list instances (and describe tags in them if
you do that )
On 01/11/2013 3:37 AM, "Gabriel Holder" gavys...@gmail.com wrote:
Update:
I managed to get it to work. I did not realize how finicky the YML file
is with the correct spacing.
I needed to add an EIP so it could check my AWS credentials on one of
their sites.

Can I create a NAT instance to fix this? I don't want these servers
having a public IP.

On Thursday, October 31, 2013 10:54:34 AM UTC-4, Gabriel Holder wrote:
This is what I've added:
cloud:
aws:
access_key:
secret_key:
discovery:
type: ec2
groups: SECURITY-GROUP
Unfortunately, they still don't see each other

On Thursday, October 31, 2013 3:36:56 AM UTC-4, Norberto Meijome wrote:

It's all in the docs for the ec2 plugin.
On 31/10/2013 12:51 PM, "Gabriel Holder" gavys...@gmail.com wrote:

You mean on the bottom of the file where I add my keys? I added those
already
How can I tag by security groups?

On Wednesday, October 30, 2013 9:42:23 PM UTC-4, Norberto Meijome
wrote:

It seems you didn't set discovery type ec2. And obviously u need to
provide API keys...
And you should also tell it which filter to use ( ie by tag or
security group..otherwise it will attempt to connect to any of your
hosts...which may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" gavys...@gmail.com wrote:

I have two ES servers in AWS, I have security groups set to allow
all traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they don't
discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ]
[Burstarr] version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:
50**:20Z]
[2013-10-30 16:11:12,703][INFO ][node ]
[Burstarr] initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ]
[Burstarr] loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][discovery**.zen.ping.unicast]
[Burstarr] using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][discovery**.zen ]
[Burstarr] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][discovery**.zen.elect ]
[Burstarr] using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][discovery**.zen.fd ]
[Burstarr] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][discovery**.zen.fd ]
[Burstarr] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.local ]
[Burstarr] using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.**recovery ]
[Burstarr] using max_bytes_per_sec[20mb], concurrent_streams [3],
file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and
compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.local.state.meta ]
[Burstarr] using gateway.local.auto_import_**dangled [YES],
with gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.meta ]
[Burstarr] took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.**local.state.shards]
[Burstarr] took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ]
[Burstarr] initialized
[2013-10-30 16:11:16,747][INFO ][node ]
[Burstarr] starting ...
[2013-10-30 16:11:16,882][INFO ][transport ]
[Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][discovery ]
[Burstarr] waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][discovery.zen ]
[Burstarr] full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][discovery.zen ]
[Burstarr] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ]
[Burstarr] new_master [Burstarr][pRsc595IQfyXn24N27t**
bZw][inet[/10.20.4.158:9300]], reason: zen-disco-join
(elected_as_master)
[2013-10-30 16:11:19,977][TRACE][discovery ]
[Burstarr] initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ]
[Burstarr] elasticsearch/**pRsc595IQfyXn24N****27tbZw
[2013-10-30 16:11:20,012][INFO ][http ]
[Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ]
[Burstarr] started
[2013-10-30 16:11:20,025][INFO ][gateway ]
[Burstarr] recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/**grou**
ps/opt_out https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/**grou**ps/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

gzion7 · November 1, 2013, 2:25am

I'll test 9300 when I get the chance.
When I didn't have the instances assigned an EIP they would generate an
error saying unable to connect to "ec2.aws.." or something (I don't have
the exact error at hand).
I believe the EIP allows it to check my AWS credentials and validate them
and then allows the nodes to see each other.

These servers ideally should only be able to access what is specified in
the security groups.

This is my first time having to actually specify the discovery.ec2 option
to my apologies for the confusion.

In my old setup we had a NAT instance and didn't need the discovery:ec2
option.

On Thursday, October 31, 2013 10:15:58 PM UTC-4, Norberto Meijome wrote:

9200 is http API port. Clustering goes over 9300 ( by default)
I mean the API key you use has to be able to get the information it needs
to find servers to speak to. It does this by connecting to AWS API and
listing instances filtered by what you tell it in the config. But yes, you
should be able to get this information manually from the CLI , as a
debugging step.
On 01/11/2013 9:39 AM, "Gabriel Holder" <gavys...@gmail.com <javascript:>>
wrote:
I can telnet just fine to 9200, I'll have to test 9300.

These are within the same region.

Do you mean running ec2-describe-instances from the command line?

I think the reason why I need a public IP is because it tries to validate
my AWS credentials on their site. This could just be because at this moment
I don't have a
NAT instance setup so I had to actually add the discovery: ec2...etc part.

In my previous setup I didn't need to specify discovery:ec2 part in my
config with the NAT instance. They were able to see each other simply
because of the AWS Plugin.

On Thursday, October 31, 2013 6:22:29 PM UTC-4, Norberto Meijome wrote:
I don't see why you would need a public IP... Just make sure:

your instances can reach each other via the protocol used for
clustering (TCP/9300 by default)

point above includes instances which may appear external to your
cluster.. Across regions?..

you API key can properly list instances (and describe tags in them if
you do that )
On 01/11/2013 3:37 AM, "Gabriel Holder" gavys...@gmail.com wrote:
Update:
I managed to get it to work. I did not realize how finicky the YML file
is with the correct spacing.
I needed to add an EIP so it could check my AWS credentials on one of
their sites.

Can I create a NAT instance to fix this? I don't want these servers
having a public IP.

On Thursday, October 31, 2013 10:54:34 AM UTC-4, Gabriel Holder wrote:
This is what I've added:
cloud:
aws:
access_key:
secret_key:
discovery:
type: ec2
groups: SECURITY-GROUP
Unfortunately, they still don't see each other

On Thursday, October 31, 2013 3:36:56 AM UTC-4, Norberto Meijome wrote:

It's all in the docs for the ec2 plugin.
On 31/10/2013 12:51 PM, "Gabriel Holder" gavys...@gmail.com wrote:

You mean on the bottom of the file where I add my keys? I added
those already
How can I tag by security groups?

On Wednesday, October 30, 2013 9:42:23 PM UTC-4, Norberto Meijome
wrote:

It seems you didn't set discovery type ec2. And obviously u need to
provide API keys...
And you should also tell it which filter to use ( ie by tag or
security group..otherwise it will attempt to connect to any of your
hosts...which may be many)
On 31/10/2013 7:13 AM, "Gabriel Holder" gavys...@gmail.com wrote:

I have two ES servers in AWS, I have security groups set to allow
all traffic from these 2 IPs.

I installed the aws plugin.
The only changes I've made in the yml file are the cluster name,
http.max.content.length, and path to store data.
Both nodes start perfectly fine, there are no errors but they
don't discover each other.

Here are the logs from starting ES on one of the nodes:
[2013-10-30 16:11:12,702][INFO ][node ]
[Burstarr] version[0.90.5], pid[25867], build[c8714e8/2013-09-17T12:
50**:20Z]
[2013-10-30 16:11:12,703][INFO ][node ]
[Burstarr] initializing ...
[2013-10-30 16:11:12,723][INFO ][plugins ]
[Burstarr] loaded [cloud-aws], sites
[2013-10-30 16:11:15,199][DEBUG][discovery**.zen.ping.unicast]
[Burstarr] using initial hosts , with concurrent_connects [10]
[2013-10-30 16:11:15,202][DEBUG][discovery**.zen ]
[Burstarr] using ping.timeout [3s], master_election.filter_client [true],
master_election.filter_data [false]
[2013-10-30 16:11:15,203][DEBUG][discovery**.zen.elect ]
[Burstarr] using minimum_master_nodes [-1]
[2013-10-30 16:11:15,205][DEBUG][discovery**.zen.fd ]
[Burstarr] [master] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:15,211][DEBUG][discovery**.zen.fd ]
[Burstarr] [node ] uses ping_interval [1s], ping_timeout [30s],
ping_retries [3]
[2013-10-30 16:11:16,344][DEBUG][gateway.local ]
[Burstarr] using initial_shards [quorum], list_timeout [30s]
[2013-10-30 16:11:16,533][DEBUG][indices.**recovery ]
[Burstarr] using max_bytes_per_sec[20mb], concurrent_streams [3],
file_chunk_size [512kb], translog_size [512kb], translog_ops [1000], and
compress [true]
[2013-10-30 16:11:16,667][DEBUG][gateway.local.state.meta ]
[Burstarr] using gateway.local.auto_import_**dangled [YES],
with gateway.local.dangling_timeout [2h]
[2013-10-30 16:11:16,692][DEBUG][gateway.local.state.meta ]
[Burstarr] took 25ms to load state
[2013-10-30 16:11:16,692][DEBUG][gateway.**local.state.shards]
[Burstarr] took 0s to load started shards state
[2013-10-30 16:11:16,746][INFO ][node ]
[Burstarr] initialized
[2013-10-30 16:11:16,747][INFO ][node ]
[Burstarr] starting ...
[2013-10-30 16:11:16,882][INFO ][transport ]
[Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
{inet[/10.20.4.158:9300]}
[2013-10-30 16:11:16,899][TRACE][discovery ]
[Burstarr] waiting for 30s for the initial state to be set by the discovery
[2013-10-30 16:11:19,905][TRACE][discovery.zen ]
[Burstarr] full ping responses: {none}
[2013-10-30 16:11:19,905][DEBUG][discovery.zen ]
[Burstarr] filtered ping responses: (filter_client[true],
filter_data[false]) {none}
[2013-10-30 16:11:19,912][INFO ][cluster.service ]
[Burstarr] new_master [Burstarr][pRsc595IQfyXn24N27t**
bZw][inet[/10.20.4.158:9300]], reason: zen-disco-join
(elected_as_master)
[2013-10-30 16:11:19,977][TRACE][discovery ]
[Burstarr] initial state set from discovery
[2013-10-30 16:11:19,977][INFO ][discovery ]
[Burstarr] elasticsearch/**pRsc595IQfyXn24N****27tbZw
[2013-10-30 16:11:20,012][INFO ][http ]
[Burstarr] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
{inet[/10.20.4.158:9200]}
[2013-10-30 16:11:20,012][INFO ][node ]
[Burstarr] started
[2013-10-30 16:11:20,025][INFO ][gateway ]
[Burstarr] recovered [0] indices into cluster_state

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/**grou**
ps/opt_out https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/**grou**ps/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Topic		Replies	Views
Instances wont see each other on AWS Elasticsearch	10	2087	July 6, 2017
Can't get Nodes to join AWS cluster Elasticsearch	3	579	July 6, 2017
Cluster nodes don't see each other Elasticsearch	5	2925	July 5, 2017
Elasticsearch 6.2.4 nodes can’t discover each other in AWS Elasticsearch	2	1006	July 6, 2018
Two nodes cluster - can't detect each other Elasticsearch	3	661	March 12, 2020

AWS nodes don't discover each other

Related topics