# Ec2 discovery stopped working!

**URL:** https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036
**Category:** Elasticsearch
**Created:** [August 1, 2013, 9:23pm UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036 "2013-08-01T21:23:12Z")
**Posts on this page:** 16
**Page:** 1

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [August 1, 2013, 9:23pm UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/1 "2013-08-01T21:23:12Z")

</div>

Since yesterday, new client-only nodes I bring up can no longer discover  
existing nodes on different machines ("waited for 30s and no initial state  
was set by the discovery").

What's odd is that I haven't changed anything in my setup (elasticsearch  
0.90.2, elasticsearch-cloud-aws 1.12.0, S3 gateway), and the machines are  
in the same security group and can see each other (i.e. I can connect to  
port 9200).

I've seen this problem previously, but only sporadically.

If I bring up a 2nd data node, rather than just a client node, the 1st data  
node appears to get stuck in a "auto expanded replicas" loop and eventually  
runs out of memory...

2013-08-01 10:47:41,620 [INFO] cluster.metadata - [Blindside] updating  
number\_of\_replicas to [1] for indices [xyz]  
2013-08-01 10:47:41,640 [INFO] cluster.metadata - [Blindside] [xyz] auto  
expanded replicas to [1]  
...

I tried setting "discovery.ec2.ping\_timeout" to "15s", but that doesn't  
make any difference. Other ideas?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [August 5, 2013, 11:43pm UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/2 "2013-08-05T23:43:06Z")

</div>

Should add that I can telnet from the client machine to port 9300 on the  
machine running elasticsearch.

What else could I check?

On Thursday, August 1, 2013 2:23:12 PM UTC-7, Eric Jain wrote:

> Since yesterday, new client-only nodes I bring up can no longer discover  
> existing nodes on different machines ("waited for 30s and no initial state  
> was set by the discovery").
> 
> What's odd is that I haven't changed anything in my setup (elasticsearch  
> 0.90.2, elasticsearch-cloud-aws 1.12.0, S3 gateway), and the machines are  
> in the same security group and can see each other (i.e. I can connect to  
> port 9200).
> 
> I've seen this problem previously, but only sporadically.
> 
> If I bring up a 2nd data node, rather than just a client node, the 1st  
> data node appears to get stuck in a "auto expanded replicas" loop and  
> eventually runs out of memory...
> 
> 2013-08-01 10:47:41,620 [INFO] cluster.metadata - [Blindside] updating  
> number\_of\_replicas to [1] for indices [xyz]  
> 2013-08-01 10:47:41,640 [INFO] cluster.metadata - [Blindside] [xyz] auto  
> expanded replicas to [1]  
> ...
> 
> I tried setting "discovery.ec2.ping\_timeout" to "15s", but that doesn't  
> make any difference. Other ideas?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [August 9, 2013, 11:24pm UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/3 "2013-08-09T23:24:48Z")

</div>

Reposted here:

> <https://stackoverflow.com/questions/18157096/unreliable-discovery-for-elasticsearch-nodes-on-ec2>

On Monday, August 5, 2013 4:43:06 PM UTC-7, Eric Jain wrote:

> Should add that I can telnet from the client machine to port 9300 on the  
> machine running elasticsearch.
> 
> What else could I check?
> 
> On Thursday, August 1, 2013 2:23:12 PM UTC-7, Eric Jain wrote:
> 
> > Since yesterday, new client-only nodes I bring up can no longer discover  
> > existing nodes on different machines ("waited for 30s and no initial state  
> > was set by the discovery").
> > 
> > What's odd is that I haven't changed anything in my setup (elasticsearch  
> > 0.90.2, elasticsearch-cloud-aws 1.12.0, S3 gateway), and the machines are  
> > in the same security group and can see each other (i.e. I can connect to  
> > port 9200).
> > 
> > I've seen this problem previously, but only sporadically.
> > 
> > If I bring up a 2nd data node, rather than just a client node, the 1st  
> > data node appears to get stuck in a "auto expanded replicas" loop and  
> > eventually runs out of memory...
> > 
> > 2013-08-01 10:47:41,620 [INFO] cluster.metadata - [Blindside] updating  
> > number\_of\_replicas to [1] for indices [xyz]  
> > 2013-08-01 10:47:41,640 [INFO] cluster.metadata - [Blindside] [xyz] auto  
> > expanded replicas to [1]  
> > ...
> > 
> > I tried setting "discovery.ec2.ping\_timeout" to "15s", but that doesn't  
> > make any difference. Other ideas?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![baniu\_yao](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/baniu_yao/32/2168_2.png) [@baniu\_yao](https://discuss.elastic.co/u/baniu_yao)
#### Post date: [August 12, 2013, 8:27am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/4 "2013-08-12T08:27:08Z")

</div>

Well, if your node in not in the same C address IP (for example, master:  
10.1.1.1, node: 10.1.2.1), you need to specify the params like this:  
'discovery.zen.ping.unicast.hosts: ["10.19.1.1"]  
'

在 2013年8月2日星期五UTC+8上午5时23分12秒，Eric Jain写道：

> Since yesterday, new client-only nodes I bring up can no longer discover  
> existing nodes on different machines ("waited for 30s and no initial state  
> was set by the discovery").
> 
> What's odd is that I haven't changed anything in my setup (elasticsearch  
> 0.90.2, elasticsearch-cloud-aws 1.12.0, S3 gateway), and the machines are  
> in the same security group and can see each other (i.e. I can connect to  
> port 9200).
> 
> I've seen this problem previously, but only sporadically.
> 
> If I bring up a 2nd data node, rather than just a client node, the 1st  
> data node appears to get stuck in a "auto expanded replicas" loop and  
> eventually runs out of memory...
> 
> 2013-08-01 10:47:41,620 [INFO] cluster.metadata - [Blindside] updating  
> number\_of\_replicas to [1] for indices [xyz]  
> 2013-08-01 10:47:41,640 [INFO] cluster.metadata - [Blindside] [xyz] auto  
> expanded replicas to [1]  
> ...
> 
> I tried setting "discovery.ec2.ping\_timeout" to "15s", but that doesn't  
> make any difference. Other ideas?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [August 12, 2013, 7:00pm UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/5 "2013-08-12T19:00:13Z")

</div>

On Mon, Aug 12, 2013 at 1:27 AM, 姚仁捷 [baniu.yao@gmail.com](mailto:baniu.yao@gmail.com) wrote:

> Well, if your node in not in the same C address IP (for example, master:  
> 10.1.1.1, node: 10.1.2.1), you need to specify the params like this:  
> 'discovery.zen.ping.unicast.hosts: ["10.19.1.1"]  
> '

Right now I have a healthy cluster with nodes in different A networks  
(e.g. 23.20.43.x and 54.221.47.x), so I don't think the  
'discovery.zen.ping.unicast.hosts' parameter is required when using  
the cloud-aws plugin. Other ideas?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [October 3, 2013, 8:00am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/6 "2013-10-03T08:00:37Z")

</div>

Just ran into this again (elasticsearch 0.90.3, elasticsearch-cloud-aws  
1.14.0): New nodes ignore existing nodes. The only solution appears to be  
to shut down the entire cluster first ☹

Looking at the (TRACE-level) logs, the old nodes do seem to be discovered  
at first, but then the new node elects itself as master,  
[http://localhost:9200/\_cluster/health](http://localhost:9200/_cluster/health) reports a single data node, and  
shards are restored from S3!

Any ideas (other than don't use the ec2 discovery mechanism)?

On Thursday, August 1, 2013 2:23:12 PM UTC-7, Eric Jain wrote:

> Since yesterday, new client-only nodes I bring up can no longer discover  
> existing nodes on different machines ("waited for 30s and no initial state  
> was set by the discovery").
> 
> What's odd is that I haven't changed anything in my setup (elasticsearch  
> 0.90.2, elasticsearch-cloud-aws 1.12.0, S3 gateway), and the machines are  
> in the same security group and can see each other (i.e. I can connect to  
> port 9200).
> 
> I've seen this problem previously, but only sporadically.
> 
> If I bring up a 2nd data node, rather than just a client node, the 1st  
> data node appears to get stuck in a "auto expanded replicas" loop and  
> eventually runs out of memory...
> 
> 2013-08-01 10:47:41,620 [INFO] cluster.metadata - [Blindside] updating  
> number\_of\_replicas to [1] for indices [xyz]  
> 2013-08-01 10:47:41,640 [INFO] cluster.metadata - [Blindside] [xyz] auto  
> expanded replicas to [1]  
> ...
> 
> I tried setting "discovery.ec2.ping\_timeout" to "15s", but that doesn't  
> make any difference. Other ideas?

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [December 4, 2013, 2:56am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/7 "2013-12-04T02:56:38Z")

</div>

On Thursday, October 3, 2013 1:00:37 AM UTC-7, Eric Jain wrote:

> Just ran into this again (elasticsearch 0.90.3, elasticsearch-cloud-aws  
> 1.14.0): New nodes ignore existing nodes. The only solution appears to be  
> to shut down the entire cluster first ☹

For the record, this problem resurfaced again after a few weeks (now using  
elasticsearch 0.19.7 and elasticsearch-cloud-aws 1.16.0)--I still don't  
know what the cause is.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dd290e8e-b1f6-4921-9668-7aeadc5af074%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dd290e8e-b1f6-4921-9668-7aeadc5af074%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Andrej](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Andrej](https://discuss.elastic.co/u/Andrej)
#### Post date: [December 4, 2013, 11:49am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/8 "2013-12-04T11:49:07Z")

</div>

Reading the docs it says that aws plugin 1.16.0 is for elasticsearch 0.90.4  
and higher. I would not expect it to run with 0.19.7.

Greets  
Andrej

Am Mittwoch, 4. Dezember 2013 03:56:38 UTC+1 schrieb Eric Jain:

> On Thursday, October 3, 2013 1:00:37 AM UTC-7, Eric Jain wrote:
> 
> > Just ran into this again (elasticsearch 0.90.3, elasticsearch-cloud-aws  
> > 1.14.0): New nodes ignore existing nodes. The only solution appears to be  
> > to shut down the entire cluster first ☹
> 
> For the record, this problem resurfaced again after a few weeks (now using  
> elasticsearch 0.19.7 and elasticsearch-cloud-aws 1.16.0)--I still don't  
> know what the cause is.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c542cd53-6f1d-46ab-9ff3-0d5ce15c6c7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c542cd53-6f1d-46ab-9ff3-0d5ce15c6c7e%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [December 4, 2013, 2:11pm UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/9 "2013-12-04T14:11:45Z")

</div>

I think Eric is using 0.90.7 and not 0.19.7…

🙂

--  
David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com)  
@dadoonet | @elasticsearchfr

Le 4 décembre 2013 at 12:49:11, Andrej Rosenheinrich (andrej.rosenheinrich@unister.de) a écrit:

Reading the docs it says that aws plugin 1.16.0 is for elasticsearch 0.90.4 and higher. I would not expect it to run with 0.19.7.

Greets  
Andrej

Am Mittwoch, 4. Dezember 2013 03:56:38 UTC+1 schrieb Eric Jain:  
On Thursday, October 3, 2013 1:00:37 AM UTC-7, Eric Jain wrote:  
Just ran into this again (elasticsearch 0.90.3, elasticsearch-cloud-aws 1.14.0): New nodes ignore existing nodes. The only solution appears to be to shut down the entire cluster first ☹

For the record, this problem resurfaced again after a few weeks (now using elasticsearch 0.19.7 and elasticsearch-cloud-aws 1.16.0)--I still don't know what the cause is.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c542cd53-6f1d-46ab-9ff3-0d5ce15c6c7e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c542cd53-6f1d-46ab-9ff3-0d5ce15c6c7e%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.529f3821.4f4ef005.bd3d%40MacBook-Air-de-David.local](https://groups.google.com/d/msgid/elasticsearch/etPan.529f3821.4f4ef005.bd3d%40MacBook-Air-de-David.local).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [December 4, 2013, 6:12pm UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/10 "2013-12-04T18:12:47Z")

</div>

On Wed, Dec 4, 2013 at 6:11 AM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:

> I think Eric is using 0.90.7 and not 0.19.7…

Yes, sorry for the confusion; I wish the problem was that simple 🙂

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHte5%2B%2BapdgscPomGTnwSAAXTb\_w-eJeeebXTxMEoaQkcu90%2Bw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHte5%2B%2BapdgscPomGTnwSAAXTb_w-eJeeebXTxMEoaQkcu90%2Bw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [December 5, 2013, 4:07am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/11 "2013-12-05T04:07:50Z")

</div>

On Tuesday, December 3, 2013 6:56:38 PM UTC-8, Eric Jain wrote:

> For the record, this problem resurfaced again after a few weeks (now using  
> elasticsearch 0.19.7 and elasticsearch-cloud-aws 1.16.0)--I still don't  
> know what the cause is.

Here is the log file (leading up to the the  
"org.elasticsearch.discovery.MasterNotDiscoveredException: waited for  
[30s]" exception) :

[application.log · GitHub](https://gist.github.com/ejain/4b2a57f4ff4cbaea0dec)

There are two other machines running, both with a client-only and a data  
node, and there's nothing obviously wrong with the cluster:

curl -XGET '[http://10.10.209.204:9200/\_cluster/health?pretty=true](http://10.10.209.204:9200/_cluster/health?pretty=true)'  
{  
"cluster\_name" : "prod-39",  
"status" : "green",  
"timed\_out" : false,  
"number\_of\_nodes" : 4,  
"number\_of\_data\_nodes" : 2,  
"active\_primary\_shards" : 226,  
"active\_shards" : 452,  
"relocating\_shards" : 0,  
"initializing\_shards" : 0,  
"unassigned\_shards" : 0  
}

But as mentioned previously, the only way to recover from this situation is  
to shut down all nodes (or copy the data and start a new cluster).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7cd0c53a-0fa8-456a-9b3e-f4609d3eb3db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7cd0c53a-0fa8-456a-9b3e-f4609d3eb3db%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 5, 2013, 4:09am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/12 "2013-12-05T04:09:25Z")

</div>

It might be worth upgrading to 0.90.X, from what I have seen there was some  
major improvements in discovery.

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 5 December 2013 15:07, Eric Jain [eric.jain@gmail.com](mailto:eric.jain@gmail.com) wrote:

> On Tuesday, December 3, 2013 6:56:38 PM UTC-8, Eric Jain wrote:
> 
> > For the record, this problem resurfaced again after a few weeks (now  
> > using elasticsearch 0.19.7 and elasticsearch-cloud-aws 1.16.0)--I still  
> > don't know what the cause is.
> 
> Here is the log file (leading up to the the  
> "org.elasticsearch.discovery.MasterNotDiscoveredException: waited for  
> [30s]" exception) :
> 
> [application.log · GitHub](https://gist.github.com/ejain/4b2a57f4ff4cbaea0dec)
> 
> There are two other machines running, both with a client-only and a data  
> node, and there's nothing obviously wrong with the cluster:
> 
> curl -XGET '[http://10.10.209.204:9200/\_cluster/health?pretty=true](http://10.10.209.204:9200/_cluster/health?pretty=true)'  
> {  
> "cluster\_name" : "prod-39",  
> "status" : "green",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 4,  
> "number\_of\_data\_nodes" : 2,  
> "active\_primary\_shards" : 226,  
> "active\_shards" : 452,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 0,  
> "unassigned\_shards" : 0  
> }
> 
> But as mentioned previously, the only way to recover from this situation  
> is to shut down all nodes (or copy the data and start a new cluster).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/7cd0c53a-0fa8-456a-9b3e-f4609d3eb3db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7cd0c53a-0fa8-456a-9b3e-f4609d3eb3db%40googlegroups.com)  
> .  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bG\_Y%3DBVzAh7quxi3Jk5SOYVBhFJk4Go103x22nR1a8kw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bG_Y%3DBVzAh7quxi3Jk5SOYVBhFJk4Go103x22nR1a8kw%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [December 5, 2013, 4:14am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/13 "2013-12-05T04:14:05Z")

</div>

On Wed, Dec 4, 2013 at 8:09 PM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) wrote:

> It might be worth upgrading to 0.90.X, from what I have seen there was some  
> major improvements in discovery.

As mentioned above, I am in fact using the latest (production) version  
of both elasticsearch (0.90.7) and the elasticsearch-cloud-aws plugin  
(1.16.0).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BJ-vOhu%3DDpyt3F9d\_tnQGiLisKjmL-NsXcRpDwDnV5Mhg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BJ-vOhu%3DDpyt3F9d_tnQGiLisKjmL-NsXcRpDwDnV5Mhg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [December 5, 2013, 4:16am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/14 "2013-12-05T04:16:44Z")

</div>

Ah, the quote with "now using elasticsearch 0.19.7" threw me.

Regards,  
Mark Walkom

Infrastructure Engineer  
Campaign Monitor  
email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
web: [www.campaignmonitor.com](http://www.campaignmonitor.com)

On 5 December 2013 15:14, Eric Jain [eric.jain@gmail.com](mailto:eric.jain@gmail.com) wrote:

> On Wed, Dec 4, 2013 at 8:09 PM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com)  
> wrote:
> 
> > It might be worth upgrading to 0.90.X, from what I have seen there was  
> > some  
> > major improvements in discovery.
> 
> As mentioned above, I am in fact using the latest (production) version  
> of both elasticsearch (0.90.7) and the elasticsearch-cloud-aws plugin  
> (1.16.0).
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BJ-vOhu%3DDpyt3F9d\_tnQGiLisKjmL-NsXcRpDwDnV5Mhg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BJ-vOhu%3DDpyt3F9d_tnQGiLisKjmL-NsXcRpDwDnV5Mhg%40mail.gmail.com)  
> .  
> For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YFfqvMz39WOrnxLBEB8c3CVzhRdt-s5AUE1579rhHogg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YFfqvMz39WOrnxLBEB8c3CVzhRdt-s5AUE1579rhHogg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)
#### Post date: [December 5, 2013, 4:19am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/15 "2013-12-05T04:19:39Z")

</div>

On Wed, Dec 4, 2013 at 8:16 PM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) wrote:

> Ah, the quote with "now using elasticsearch 0.19.7" threw me.

Yeah, I shouldn't have quoted my own typo again 🙂

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKn9a9vdL33v-vRU9jWbj5ZzyHma8%3DiYh53kQKLaJQ0RQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKn9a9vdL33v-vRU9jWbj5ZzyHma8%3DiYh53kQKLaJQ0RQ%40mail.gmail.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:03am UTC](https://discuss.elastic.co/t/ec2-discovery-stopped-working/13036/16 "2017-07-06T02:03:11Z")

</div>


