# ES bugs in 0.20.4 and 0.20.5 cause shards allocation failure and stuck in initializing state

**URL:** <https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915>\
**Category:** Elasticsearch\
**Created:** [February 27, 2013, 12:09am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915 "2013-02-27T00:09:00Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [February 27, 2013, 12:09am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/1 "2013-02-27T00:09:00Z")

</div>

Hi, guys:  
These days through the test, I found ES bugs in 0.20.4 and 0.20.5 which  
cause shards allocation failure and stuck in initializing state.  
The following are my test steps:

1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3 nodes  
are master nodes, 2 nodes are load balancer, 15 nodes are data nodes
2. After the cluster is up, I tried to create some empty indices for  
example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
index-2013-03-01, etc  
But some shards stuck in initializing status for long time.  
{  
"cluster\_name" : "es-test",  
"status" : "yellow",  
"timed\_out" : false,  
"number\_of\_nodes" : 20,  
"number\_of\_data\_nodes" : 15,  
"active\_primary\_shards" : 105,  
"active\_shards" : 201,  
"relocating\_shards" : 0,  
"initializing\_shards" : 9,  
"unassigned\_shards" : 0  
}  
Moreover when I created index-2013-03-02, the cluster became to red.  
{  
"cluster\_name" : "es-test",  
"status" : "red",  
"timed\_out" : false,  
"number\_of\_nodes" : 20,  
"number\_of\_data\_nodes" : 15,  
"active\_primary\_shards" : 119,  
"active\_shards" : 228,  
"relocating\_shards" : 0,  
"initializing\_shards" : 11,  
"unassigned\_shards" : 1  
}

I set the log level to trace, and checked the logs, no error logs are  
shown. but from the logs, I can know some shards are initializing and  
unassigned.  
And I just tried 0.20.5, the same problem happened.  
But for 0.19.11, the problem disappeared. All the empty indices can be  
created successfully instantly even for some strange index names.  
So I guess ES have some bugs for 0.20.4 and 0.20.5.  
Can Kimchy or any other es experts have a check this problem?  
Thank you very much!

-Regards-  
-Dong Aihua-

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [February 27, 2013, 3:10am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/2 "2013-02-27T03:10:42Z")

</div>

By the way, I tested the single node for 0.20.4, 0.20.5, such problem  
doesn't happen.

在 2013年2月27日星期三UTC+9上午9时09分00秒，jackiedong写道：

> Hi, guys:  
> These days through the test, I found ES bugs in 0.20.4 and 0.20.5 which  
> cause shards allocation failure and stuck in initializing state.  
> The following are my test steps:
> 
> 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3 nodes  
> are master nodes, 2 nodes are load balancer, 15 nodes are data nodes
> 2. After the cluster is up, I tried to create some empty indices for  
> example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> index-2013-03-01, etc  
> But some shards stuck in initializing status for long time.  
> {  
> "cluster\_name" : "es-test",  
> "status" : "yellow",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 105,  
> "active\_shards" : 201,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 9,  
> "unassigned\_shards" : 0  
> }  
> Moreover when I created index-2013-03-02, the cluster became to red.  
> {  
> "cluster\_name" : "es-test",  
> "status" : "red",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 119,  
> "active\_shards" : 228,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 11,  
> "unassigned\_shards" : 1  
> }
> 
> I set the log level to trace, and checked the logs, no error logs are  
> shown. but from the logs, I can know some shards are initializing and  
> unassigned.  
> And I just tried 0.20.5, the same problem happened.  
> But for 0.19.11, the problem disappeared. All the empty indices can be  
> created successfully instantly even for some strange index names.  
> So I guess ES have some bugs for 0.20.4 and 0.20.5.  
> Can Kimchy or any other es experts have a check this problem?  
> Thank you very much!
> 
> -Regards-  
> -Dong Aihua-

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [February 27, 2013, 10:51am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/3 "2013-02-27T10:51:49Z")

</div>

On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:

> Hi, guys:  
> These days through the test, I found ES bugs in 0.20.4 and 0.20.5  
> which cause shards allocation failure and stuck in initializing state.  
> The following are my test steps:
> 
> 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3  
> nodes are master nodes, 2 nodes are load balancer, 15 nodes are data  
> nodes
> 2. After the cluster is up, I tried to create some empty indices for  
> example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> index-2013-03-01, etc  
> But some shards stuck in initializing status for long time.

Please can you open an issue on github, with a full recreation of what  
you need to do to recreate this problem, plus all the logs from all of  
the nodes.

ta

Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\
**Post date:** [March 4, 2013, 8:47am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/4 "2013-03-04T08:47:52Z")

</div>

Hey,

thanks for testing this. I can see some exception during recovery when  
starting a replica which causes one of the machines to wait for an answer  
but it doesn't come back and it doesn't seem to get notified that the  
connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
30s"  
So we can see if this happens just because of the closed connections here?

simon

On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:

> Hi, Clint:  
> I already opened an issue  
> [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> Today I tested the latest the ES version 0.90.0.Beta1, it has the same  
> problem.  
> My test configure is as following:  
> 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the leader; 214  
> and 215 are load balancer; other 15 nodes are data nodes.  
> My test step is as following:
> 
> 1. after the cluster is up, I created an empty index: test1  
> curl -XPUT 10.96.250.214:10200/test1  
> {"ok":true,"acknowledged":true}  
> curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> {  
> "cluster\_name" : "es-test-0-90-0-beta1",  
> "status" : "yellow",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 15,  
> "active\_shards" : 27,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 3,  
> "unassigned\_shards" : 0  
> }  
> The cluster stayed in this state for long time.
> 2. Then I created another empty index: abcd1234  
> curl -XPUT 10.96.250.214:10200/abcd1234  
> {"ok":true,"acknowledged":false}  
> curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> {  
> "cluster\_name" : "es-test-0-90-0-beta1",  
> "status" : "red",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 29,  
> "active\_shards" : 53,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 6,  
> "unassigned\_shards" : 1  
> The cluster stayed in this state for long time
> 3. Then I created one more empty index:  
> curl -XPUT 10.96.250.214:10200/1234abcd  
> {"ok":true,"acknowledged":false}  
> curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> {  
> "cluster\_name" : "es-test-0-90-0-beta1",  
> "status" : "red",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 43,  
> "active\_shards" : 78,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 6,  
> "unassigned\_shards" : 6  
> }  
> The cluster stayed in this state for long time.  
> You can refer the detail logs from 20 nodes in the attachments.  
> Thank you.
> 
> -Regards-  
> -Dong Aihua-
> 
> 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> 
> > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > 
> > > Hi, guys:  
> > > These days through the test, I found ES bugs in 0.20.4 and 0.20.5  
> > > which cause shards allocation failure and stuck in initializing state.  
> > > The following are my test steps:
> > > 
> > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3  
> > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are data  
> > > nodes
> > > 2. After the cluster is up, I tried to create some empty indices for  
> > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > index-2013-03-01, etc  
> > > But some shards stuck in initializing status for long time.
> > 
> > Please can you open an issue on github, with a full recreation of what  
> > you need to do to recreate this problem, plus all the logs from all of  
> > the nodes.
> > 
> > ta
> > 
> > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 5, 2013, 5:21am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/5 "2013-03-05T05:21:42Z")

</div>

Hi, Simon:  
I tested the ES version 0.90.0.Beta1 again with the setting  
indices.recovery.internal\_action\_timeout: 30s.  
The same problem happened the again.  
The configuration is same as before. The following are my test steps:

1. curl -XPUT 10.96.250.214:10200/test1  
{"ok":true,"acknowledged":true}  
{  
"cluster\_name" : "es-test2-0-90-0-beta1",  
"status" : "yellow",  
"timed\_out" : false,  
"number\_of\_nodes" : 20,  
"number\_of\_data\_nodes" : 15,  
"active\_primary\_shards" : 15,  
"active\_shards" : 28,  
"relocating\_shards" : 0,  
"initializing\_shards" : 2,  
"unassigned\_shards" : 0  
}  
The cluster stayed in this state for long time
2. curl -XPUT 10.96.250.214:10200/1234abcd  
{"ok":true,"acknowledged":true}  
{  
"cluster\_name" : "es-test2-0-90-0-beta1",  
"status" : "yellow",  
"timed\_out" : false,  
"number\_of\_nodes" : 20,  
"number\_of\_data\_nodes" : 15,  
"active\_primary\_shards" : 30,  
"active\_shards" : 57,  
"relocating\_shards" : 0,  
"initializing\_shards" : 3,  
"unassigned\_shards" : 0  
}
3. curl -XPUT 10.96.250.214:10200/abcd1234  
{  
"cluster\_name" : "es-test2-0-90-0-beta1",  
"status" : "yellow",  
"timed\_out" : false,  
"number\_of\_nodes" : 20,  
"number\_of\_data\_nodes" : 15,  
"active\_primary\_shards" : 45,  
"active\_shards" : 87,  
"relocating\_shards" : 0,  
"initializing\_shards" : 3,  
"unassigned\_shards" : 0  
}  
The cluster stayed in this state for long time
4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
{  
"cluster\_name" : "es-test2-0-90-0-beta1",  
"status" : "red",  
"timed\_out" : false,  
"number\_of\_nodes" : 20,  
"number\_of\_data\_nodes" : 15,  
"active\_primary\_shards" : 59,  
"active\_shards" : 112,  
"relocating\_shards" : 0,  
"initializing\_shards" : 7,  
"unassigned\_shards" : 1  
}  
The cluster stayed in this state for long time.  
The detail logs of 20 nodes are attached.  
Thank you.

-Regards-  
-Dong Aihua-

在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：

> Hey,
> 
> thanks for testing this. I can see some exception during recovery when  
> starting a replica which causes one of the machines to wait for an answer  
> but it doesn't come back and it doesn't seem to get notified that the  
> connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> 30s"  
> So we can see if this happens just because of the closed connections here?
> 
> simon
> 
> On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> 
> > Hi, Clint:  
> > I already opened an issue  
> > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > Today I tested the latest the ES version 0.90.0.Beta1, it has the same  
> > problem.  
> > My test configure is as following:  
> > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the leader;  
> > 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > My test step is as following:
> > 
> > 1. after the cluster is up, I created an empty index: test1  
> > curl -XPUT 10.96.250.214:10200/test1  
> > {"ok":true,"acknowledged":true}  
> > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > {  
> > "cluster\_name" : "es-test-0-90-0-beta1",  
> > "status" : "yellow",  
> > "timed\_out" : false,  
> > "number\_of\_nodes" : 20,  
> > "number\_of\_data\_nodes" : 15,  
> > "active\_primary\_shards" : 15,  
> > "active\_shards" : 27,  
> > "relocating\_shards" : 0,  
> > "initializing\_shards" : 3,  
> > "unassigned\_shards" : 0  
> > }  
> > The cluster stayed in this state for long time.
> > 2. Then I created another empty index: abcd1234  
> > curl -XPUT 10.96.250.214:10200/abcd1234  
> > {"ok":true,"acknowledged":false}  
> > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > {  
> > "cluster\_name" : "es-test-0-90-0-beta1",  
> > "status" : "red",  
> > "timed\_out" : false,  
> > "number\_of\_nodes" : 20,  
> > "number\_of\_data\_nodes" : 15,  
> > "active\_primary\_shards" : 29,  
> > "active\_shards" : 53,  
> > "relocating\_shards" : 0,  
> > "initializing\_shards" : 6,  
> > "unassigned\_shards" : 1  
> > The cluster stayed in this state for long time
> > 3. Then I created one more empty index:  
> > curl -XPUT 10.96.250.214:10200/1234abcd  
> > {"ok":true,"acknowledged":false}  
> > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > {  
> > "cluster\_name" : "es-test-0-90-0-beta1",  
> > "status" : "red",  
> > "timed\_out" : false,  
> > "number\_of\_nodes" : 20,  
> > "number\_of\_data\_nodes" : 15,  
> > "active\_primary\_shards" : 43,  
> > "active\_shards" : 78,  
> > "relocating\_shards" : 0,  
> > "initializing\_shards" : 6,  
> > "unassigned\_shards" : 6  
> > }  
> > The cluster stayed in this state for long time.  
> > You can refer the detail logs from 20 nodes in the attachments.  
> > Thank you.
> > 
> > -Regards-  
> > -Dong Aihua-
> > 
> > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > 
> > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > 
> > > > Hi, guys:  
> > > > These days through the test, I found ES bugs in 0.20.4 and 0.20.5  
> > > > which cause shards allocation failure and stuck in initializing state.  
> > > > The following are my test steps:
> > > > 
> > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3  
> > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are data  
> > > > nodes
> > > > 2. After the cluster is up, I tried to create some empty indices for  
> > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > index-2013-03-01, etc  
> > > > But some shards stuck in initializing status for long time.
> > > 
> > > Please can you open an issue on github, with a full recreation of what  
> > > you need to do to recreate this problem, plus all the logs from all of  
> > > the nodes.
> > > 
> > > ta
> > > 
> > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 5, 2013, 5:43am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/6 "2013-03-05T05:43:15Z")

</div>

I uploaded several times for the logs. All failed, I got the 340 error. I  
will upload the logs later.

在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：

> Hi, Simon:  
> I tested the ES version 0.90.0.Beta1 again with the setting  
> indices.recovery.internal\_action\_timeout: 30s.  
> The same problem happened the again.  
> The configuration is same as before. The following are my test steps:
> 
> 1. curl -XPUT 10.96.250.214:10200/test1  
> {"ok":true,"acknowledged":true}  
> {  
> "cluster\_name" : "es-test2-0-90-0-beta1",  
> "status" : "yellow",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 15,  
> "active\_shards" : 28,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 2,  
> "unassigned\_shards" : 0  
> }  
> The cluster stayed in this state for long time
> 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> {"ok":true,"acknowledged":true}  
> {  
> "cluster\_name" : "es-test2-0-90-0-beta1",  
> "status" : "yellow",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 30,  
> "active\_shards" : 57,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 3,  
> "unassigned\_shards" : 0  
> }
> 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> {  
> "cluster\_name" : "es-test2-0-90-0-beta1",  
> "status" : "yellow",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 45,  
> "active\_shards" : 87,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 3,  
> "unassigned\_shards" : 0  
> }  
> The cluster stayed in this state for long time
> 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> {  
> "cluster\_name" : "es-test2-0-90-0-beta1",  
> "status" : "red",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 59,  
> "active\_shards" : 112,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 7,  
> "unassigned\_shards" : 1  
> }  
> The cluster stayed in this state for long time.  
> The detail logs of 20 nodes are attached.  
> Thank you.
> 
> -Regards-  
> -Dong Aihua-
> 
> 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> 
> > Hey,
> > 
> > thanks for testing this. I can see some exception during recovery when  
> > starting a replica which causes one of the machines to wait for an answer  
> > but it doesn't come back and it doesn't seem to get notified that the  
> > connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > 30s"  
> > So we can see if this happens just because of the closed connections here?
> > 
> > simon
> > 
> > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > 
> > > Hi, Clint:  
> > > I already opened an issue  
> > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > Today I tested the latest the ES version 0.90.0.Beta1, it has the  
> > > same problem.  
> > > My test configure is as following:  
> > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the leader;  
> > > 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > My test step is as following:
> > > 
> > > 1. after the cluster is up, I created an empty index: test1  
> > > curl -XPUT 10.96.250.214:10200/test1  
> > > {"ok":true,"acknowledged":true}  
> > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > {  
> > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > "status" : "yellow",  
> > > "timed\_out" : false,  
> > > "number\_of\_nodes" : 20,  
> > > "number\_of\_data\_nodes" : 15,  
> > > "active\_primary\_shards" : 15,  
> > > "active\_shards" : 27,  
> > > "relocating\_shards" : 0,  
> > > "initializing\_shards" : 3,  
> > > "unassigned\_shards" : 0  
> > > }  
> > > The cluster stayed in this state for long time.
> > > 2. Then I created another empty index: abcd1234  
> > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > {"ok":true,"acknowledged":false}  
> > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > {  
> > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > "status" : "red",  
> > > "timed\_out" : false,  
> > > "number\_of\_nodes" : 20,  
> > > "number\_of\_data\_nodes" : 15,  
> > > "active\_primary\_shards" : 29,  
> > > "active\_shards" : 53,  
> > > "relocating\_shards" : 0,  
> > > "initializing\_shards" : 6,  
> > > "unassigned\_shards" : 1  
> > > The cluster stayed in this state for long time
> > > 3. Then I created one more empty index:  
> > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > {"ok":true,"acknowledged":false}  
> > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > {  
> > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > "status" : "red",  
> > > "timed\_out" : false,  
> > > "number\_of\_nodes" : 20,  
> > > "number\_of\_data\_nodes" : 15,  
> > > "active\_primary\_shards" : 43,  
> > > "active\_shards" : 78,  
> > > "relocating\_shards" : 0,  
> > > "initializing\_shards" : 6,  
> > > "unassigned\_shards" : 6  
> > > }  
> > > The cluster stayed in this state for long time.  
> > > You can refer the detail logs from 20 nodes in the attachments.  
> > > Thank you.
> > > 
> > > -Regards-  
> > > -Dong Aihua-
> > > 
> > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > 
> > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > 
> > > > > Hi, guys:  
> > > > > These days through the test, I found ES bugs in 0.20.4 and 0.20.5  
> > > > > which cause shards allocation failure and stuck in initializing  
> > > > > state.  
> > > > > The following are my test steps:
> > > > > 
> > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3  
> > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are data  
> > > > > nodes
> > > > > 2. After the cluster is up, I tried to create some empty indices  
> > > > > for  
> > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > index-2013-03-01, etc  
> > > > > But some shards stuck in initializing status for long time.
> > > > 
> > > > Please can you open an issue on github, with a full recreation of what  
> > > > you need to do to recreate this problem, plus all the logs from all of  
> > > > the nodes.
> > > > 
> > > > ta
> > > > 
> > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 5, 2013, 6:42am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/7 "2013-03-05T06:42:55Z")

</div>

Hi, Simon  
Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
Thank you.

-Regards-  
-Dong aihua-

在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：

> I uploaded several times for the logs. All failed, I got the 340 error. I  
> will upload the logs later.
> 
> 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> 
> > Hi, Simon:  
> > I tested the ES version 0.90.0.Beta1 again with the setting  
> > indices.recovery.internal\_action\_timeout: 30s.  
> > The same problem happened the again.  
> > The configuration is same as before. The following are my test steps:
> > 
> > 1. curl -XPUT 10.96.250.214:10200/test1  
> > {"ok":true,"acknowledged":true}  
> > {  
> > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > "status" : "yellow",  
> > "timed\_out" : false,  
> > "number\_of\_nodes" : 20,  
> > "number\_of\_data\_nodes" : 15,  
> > "active\_primary\_shards" : 15,  
> > "active\_shards" : 28,  
> > "relocating\_shards" : 0,  
> > "initializing\_shards" : 2,  
> > "unassigned\_shards" : 0  
> > }  
> > The cluster stayed in this state for long time
> > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > {"ok":true,"acknowledged":true}  
> > {  
> > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > "status" : "yellow",  
> > "timed\_out" : false,  
> > "number\_of\_nodes" : 20,  
> > "number\_of\_data\_nodes" : 15,  
> > "active\_primary\_shards" : 30,  
> > "active\_shards" : 57,  
> > "relocating\_shards" : 0,  
> > "initializing\_shards" : 3,  
> > "unassigned\_shards" : 0  
> > }
> > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > {  
> > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > "status" : "yellow",  
> > "timed\_out" : false,  
> > "number\_of\_nodes" : 20,  
> > "number\_of\_data\_nodes" : 15,  
> > "active\_primary\_shards" : 45,  
> > "active\_shards" : 87,  
> > "relocating\_shards" : 0,  
> > "initializing\_shards" : 3,  
> > "unassigned\_shards" : 0  
> > }  
> > The cluster stayed in this state for long time
> > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > {  
> > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > "status" : "red",  
> > "timed\_out" : false,  
> > "number\_of\_nodes" : 20,  
> > "number\_of\_data\_nodes" : 15,  
> > "active\_primary\_shards" : 59,  
> > "active\_shards" : 112,  
> > "relocating\_shards" : 0,  
> > "initializing\_shards" : 7,  
> > "unassigned\_shards" : 1  
> > }  
> > The cluster stayed in this state for long time.  
> > The detail logs of 20 nodes are attached.  
> > Thank you.
> > 
> > -Regards-  
> > -Dong Aihua-
> > 
> > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > 
> > > Hey,
> > > 
> > > thanks for testing this. I can see some exception during recovery when  
> > > starting a replica which causes one of the machines to wait for an answer  
> > > but it doesn't come back and it doesn't seem to get notified that the  
> > > connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > 30s"  
> > > So we can see if this happens just because of the closed connections  
> > > here?
> > > 
> > > simon
> > > 
> > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > 
> > > > Hi, Clint:  
> > > > I already opened an issue  
> > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > Today I tested the latest the ES version 0.90.0.Beta1, it has the  
> > > > same problem.  
> > > > My test configure is as following:  
> > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the leader;  
> > > > 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > My test step is as following:
> > > > 
> > > > 1. after the cluster is up, I created an empty index: test1  
> > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > {"ok":true,"acknowledged":true}  
> > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > {  
> > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > "status" : "yellow",  
> > > > "timed\_out" : false,  
> > > > "number\_of\_nodes" : 20,  
> > > > "number\_of\_data\_nodes" : 15,  
> > > > "active\_primary\_shards" : 15,  
> > > > "active\_shards" : 27,  
> > > > "relocating\_shards" : 0,  
> > > > "initializing\_shards" : 3,  
> > > > "unassigned\_shards" : 0  
> > > > }  
> > > > The cluster stayed in this state for long time.
> > > > 2. Then I created another empty index: abcd1234  
> > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > {"ok":true,"acknowledged":false}  
> > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > {  
> > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > "status" : "red",  
> > > > "timed\_out" : false,  
> > > > "number\_of\_nodes" : 20,  
> > > > "number\_of\_data\_nodes" : 15,  
> > > > "active\_primary\_shards" : 29,  
> > > > "active\_shards" : 53,  
> > > > "relocating\_shards" : 0,  
> > > > "initializing\_shards" : 6,  
> > > > "unassigned\_shards" : 1  
> > > > The cluster stayed in this state for long time
> > > > 3. Then I created one more empty index:  
> > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > {"ok":true,"acknowledged":false}  
> > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > {  
> > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > "status" : "red",  
> > > > "timed\_out" : false,  
> > > > "number\_of\_nodes" : 20,  
> > > > "number\_of\_data\_nodes" : 15,  
> > > > "active\_primary\_shards" : 43,  
> > > > "active\_shards" : 78,  
> > > > "relocating\_shards" : 0,  
> > > > "initializing\_shards" : 6,  
> > > > "unassigned\_shards" : 6  
> > > > }  
> > > > The cluster stayed in this state for long time.  
> > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > Thank you.
> > > > 
> > > > -Regards-  
> > > > -Dong Aihua-
> > > > 
> > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > 
> > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > 
> > > > > > Hi, guys:  
> > > > > > These days through the test, I found ES bugs in 0.20.4 and 0.20.5  
> > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > state.  
> > > > > > The following are my test steps:
> > > > > > 
> > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3  
> > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are data  
> > > > > > nodes
> > > > > > 2. After the cluster is up, I tried to create some empty indices  
> > > > > > for  
> > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > index-2013-03-01, etc  
> > > > > > But some shards stuck in initializing status for long time.
> > > > > 
> > > > > Please can you open an issue on github, with a full recreation of what  
> > > > > you need to do to recreate this problem, plus all the logs from all of  
> > > > > the nodes.
> > > > > 
> > > > > ta
> > > > > 
> > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 7, 2013, 2:11am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/8 "2013-03-07T02:11:57Z")

</div>

Hi, :  
Is there any clue for this problem's root cause?  
Thank you.

-Regards-  
-Dong Aihua-

在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：

> Hi, Simon  
> Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> Thank you.
> 
> -Regards-  
> -Dong aihua-
> 
> 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> 
> > I uploaded several times for the logs. All failed, I got the 340 error. I  
> > will upload the logs later.
> > 
> > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > 
> > > Hi, Simon:  
> > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > indices.recovery.internal\_action\_timeout: 30s.  
> > > The same problem happened the again.  
> > > The configuration is same as before. The following are my test steps:
> > > 
> > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > {"ok":true,"acknowledged":true}  
> > > {  
> > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > "status" : "yellow",  
> > > "timed\_out" : false,  
> > > "number\_of\_nodes" : 20,  
> > > "number\_of\_data\_nodes" : 15,  
> > > "active\_primary\_shards" : 15,  
> > > "active\_shards" : 28,  
> > > "relocating\_shards" : 0,  
> > > "initializing\_shards" : 2,  
> > > "unassigned\_shards" : 0  
> > > }  
> > > The cluster stayed in this state for long time
> > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > {"ok":true,"acknowledged":true}  
> > > {  
> > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > "status" : "yellow",  
> > > "timed\_out" : false,  
> > > "number\_of\_nodes" : 20,  
> > > "number\_of\_data\_nodes" : 15,  
> > > "active\_primary\_shards" : 30,  
> > > "active\_shards" : 57,  
> > > "relocating\_shards" : 0,  
> > > "initializing\_shards" : 3,  
> > > "unassigned\_shards" : 0  
> > > }
> > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > {  
> > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > "status" : "yellow",  
> > > "timed\_out" : false,  
> > > "number\_of\_nodes" : 20,  
> > > "number\_of\_data\_nodes" : 15,  
> > > "active\_primary\_shards" : 45,  
> > > "active\_shards" : 87,  
> > > "relocating\_shards" : 0,  
> > > "initializing\_shards" : 3,  
> > > "unassigned\_shards" : 0  
> > > }  
> > > The cluster stayed in this state for long time
> > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > {  
> > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > "status" : "red",  
> > > "timed\_out" : false,  
> > > "number\_of\_nodes" : 20,  
> > > "number\_of\_data\_nodes" : 15,  
> > > "active\_primary\_shards" : 59,  
> > > "active\_shards" : 112,  
> > > "relocating\_shards" : 0,  
> > > "initializing\_shards" : 7,  
> > > "unassigned\_shards" : 1  
> > > }  
> > > The cluster stayed in this state for long time.  
> > > The detail logs of 20 nodes are attached.  
> > > Thank you.
> > > 
> > > -Regards-  
> > > -Dong Aihua-
> > > 
> > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > 
> > > > Hey,
> > > > 
> > > > thanks for testing this. I can see some exception during recovery when  
> > > > starting a replica which causes one of the machines to wait for an answer  
> > > > but it doesn't come back and it doesn't seem to get notified that the  
> > > > connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > 30s"  
> > > > So we can see if this happens just because of the closed connections  
> > > > here?
> > > > 
> > > > simon
> > > > 
> > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > 
> > > > > Hi, Clint:  
> > > > > I already opened an issue  
> > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has the  
> > > > > same problem.  
> > > > > My test configure is as following:  
> > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the leader;  
> > > > > 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > My test step is as following:
> > > > > 
> > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > {"ok":true,"acknowledged":true}  
> > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > {  
> > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 15,  
> > > > > "active\_shards" : 27,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 3,  
> > > > > "unassigned\_shards" : 0  
> > > > > }  
> > > > > The cluster stayed in this state for long time.
> > > > > 2. Then I created another empty index: abcd1234  
> > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > {"ok":true,"acknowledged":false}  
> > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > {  
> > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > "status" : "red",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 29,  
> > > > > "active\_shards" : 53,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 6,  
> > > > > "unassigned\_shards" : 1  
> > > > > The cluster stayed in this state for long time
> > > > > 3. Then I created one more empty index:  
> > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > {"ok":true,"acknowledged":false}  
> > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > {  
> > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > "status" : "red",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 43,  
> > > > > "active\_shards" : 78,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 6,  
> > > > > "unassigned\_shards" : 6  
> > > > > }  
> > > > > The cluster stayed in this state for long time.  
> > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > Thank you.
> > > > > 
> > > > > -Regards-  
> > > > > -Dong Aihua-
> > > > > 
> > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > 
> > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > 
> > > > > > > Hi, guys:  
> > > > > > > These days through the test, I found ES bugs in 0.20.4 and 0.20.5  
> > > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > > state.  
> > > > > > > The following are my test steps:
> > > > > > > 
> > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3  
> > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are  
> > > > > > > data  
> > > > > > > nodes
> > > > > > > 2. After the cluster is up, I tried to create some empty indices  
> > > > > > > for  
> > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > index-2013-03-01, etc  
> > > > > > > But some shards stuck in initializing status for long time.
> > > > > > 
> > > > > > Please can you open an issue on github, with a full recreation of  
> > > > > > what  
> > > > > > you need to do to recreate this problem, plus all the logs from all  
> > > > > > of  
> > > > > > the nodes.
> > > > > > 
> > > > > > ta
> > > > > > 
> > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\
**Post date:** [March 7, 2013, 11:04am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/9 "2013-03-07T11:04:22Z")

</div>

So, I looked closer at the latest logs and I see a lot of disconnects going  
on giving me the impression you have some network issues. Nevertheless we  
pushed some stuff to detect these situations earlier but non of us was able  
to reproduce your issues. the only thing I can ask you for is to try again  
with latest master to see if those commits helped in any way?  
What is your setup by the way, any idea why servers disconnect all the time?

simon

On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:

> Hi, :  
> Is there any clue for this problem's root cause?  
> Thank you.
> 
> -Regards-  
> -Dong Aihua-
> 
> 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> 
> > Hi, Simon  
> > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > Thank you.
> > 
> > -Regards-  
> > -Dong aihua-
> > 
> > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > 
> > > I uploaded several times for the logs. All failed, I got the 340 error.  
> > > I will upload the logs later.
> > > 
> > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > 
> > > > Hi, Simon:  
> > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > The same problem happened the again.  
> > > > The configuration is same as before. The following are my test steps:
> > > > 
> > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > {"ok":true,"acknowledged":true}  
> > > > {  
> > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > "status" : "yellow",  
> > > > "timed\_out" : false,  
> > > > "number\_of\_nodes" : 20,  
> > > > "number\_of\_data\_nodes" : 15,  
> > > > "active\_primary\_shards" : 15,  
> > > > "active\_shards" : 28,  
> > > > "relocating\_shards" : 0,  
> > > > "initializing\_shards" : 2,  
> > > > "unassigned\_shards" : 0  
> > > > }  
> > > > The cluster stayed in this state for long time
> > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > {"ok":true,"acknowledged":true}  
> > > > {  
> > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > "status" : "yellow",  
> > > > "timed\_out" : false,  
> > > > "number\_of\_nodes" : 20,  
> > > > "number\_of\_data\_nodes" : 15,  
> > > > "active\_primary\_shards" : 30,  
> > > > "active\_shards" : 57,  
> > > > "relocating\_shards" : 0,  
> > > > "initializing\_shards" : 3,  
> > > > "unassigned\_shards" : 0  
> > > > }
> > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > {  
> > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > "status" : "yellow",  
> > > > "timed\_out" : false,  
> > > > "number\_of\_nodes" : 20,  
> > > > "number\_of\_data\_nodes" : 15,  
> > > > "active\_primary\_shards" : 45,  
> > > > "active\_shards" : 87,  
> > > > "relocating\_shards" : 0,  
> > > > "initializing\_shards" : 3,  
> > > > "unassigned\_shards" : 0  
> > > > }  
> > > > The cluster stayed in this state for long time
> > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > {  
> > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > "status" : "red",  
> > > > "timed\_out" : false,  
> > > > "number\_of\_nodes" : 20,  
> > > > "number\_of\_data\_nodes" : 15,  
> > > > "active\_primary\_shards" : 59,  
> > > > "active\_shards" : 112,  
> > > > "relocating\_shards" : 0,  
> > > > "initializing\_shards" : 7,  
> > > > "unassigned\_shards" : 1  
> > > > }  
> > > > The cluster stayed in this state for long time.  
> > > > The detail logs of 20 nodes are attached.  
> > > > Thank you.
> > > > 
> > > > -Regards-  
> > > > -Dong Aihua-
> > > > 
> > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > 
> > > > > Hey,
> > > > > 
> > > > > thanks for testing this. I can see some exception during recovery when  
> > > > > starting a replica which causes one of the machines to wait for an answer  
> > > > > but it doesn't come back and it doesn't seem to get notified that the  
> > > > > connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > 30s"  
> > > > > So we can see if this happens just because of the closed connections  
> > > > > here?
> > > > > 
> > > > > simon
> > > > > 
> > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > 
> > > > > > Hi, Clint:  
> > > > > > I already opened an issue  
> > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has the  
> > > > > > same problem.  
> > > > > > My test configure is as following:  
> > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > My test step is as following:
> > > > > > 
> > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > {"ok":true,"acknowledged":true}  
> > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > "status" : "yellow",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 15,  
> > > > > > "active\_shards" : 27,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 3,  
> > > > > > "unassigned\_shards" : 0  
> > > > > > }  
> > > > > > The cluster stayed in this state for long time.
> > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > {"ok":true,"acknowledged":false}  
> > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > "status" : "red",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 29,  
> > > > > > "active\_shards" : 53,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 6,  
> > > > > > "unassigned\_shards" : 1  
> > > > > > The cluster stayed in this state for long time
> > > > > > 3. Then I created one more empty index:  
> > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > {"ok":true,"acknowledged":false}  
> > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > "status" : "red",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 43,  
> > > > > > "active\_shards" : 78,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 6,  
> > > > > > "unassigned\_shards" : 6  
> > > > > > }  
> > > > > > The cluster stayed in this state for long time.  
> > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > Thank you.
> > > > > > 
> > > > > > -Regards-  
> > > > > > -Dong Aihua-
> > > > > > 
> > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > 
> > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > 
> > > > > > > > Hi, guys:  
> > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > 0.20.5  
> > > > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > > > state.  
> > > > > > > > The following are my test steps:
> > > > > > > > 
> > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up. 3  
> > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are  
> > > > > > > > data  
> > > > > > > > nodes
> > > > > > > > 2. After the cluster is up, I tried to create some empty indices  
> > > > > > > > for  
> > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > index-2013-03-01, etc  
> > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > 
> > > > > > > Please can you open an issue on github, with a full recreation of  
> > > > > > > what  
> > > > > > > you need to do to recreate this problem, plus all the logs from all  
> > > > > > > of  
> > > > > > > the nodes.
> > > > > > > 
> > > > > > > ta
> > > > > > > 
> > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [March 7, 2013, 9:38pm UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/10 "2013-03-07T21:38:30Z")

</div>

Hi Simon,

Which commits were they? 9a25867bfe154357165c87a7b509029ff832efa4? Curious  
to see what has changed.

I have not looked at Dong's logs, but we have also experienced nodes being  
removed from a cluster although the process is running. One possible  
culprit is unresponsiveness due to GC.

--  
Ivan

On Thu, Mar 7, 2013 at 3:04 AM, simonw [simon.willnauer@elasticsearch.com](mailto:simon.willnauer@elasticsearch.com)wrote:

> So, I looked closer at the latest logs and I see a lot of disconnects  
> going on giving me the impression you have some network issues.  
> Nevertheless we pushed some stuff to detect these situations earlier but  
> non of us was able to reproduce your issues. the only thing I can ask you  
> for is to try again with latest master to see if those commits helped in  
> any way?  
> What is your setup by the way, any idea why servers disconnect all the  
> time?
> 
> simon
> 
> On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> 
> > Hi, :  
> > Is there any clue for this problem's root cause?  
> > Thank you.
> > 
> > -Regards-  
> > -Dong Aihua-
> > 
> > 在 2013年3月5日星期二UTC+8下午2时42分55秒，\*\*Dong Aihua写道：
> > 
> > > Hi, Simon  
> > > Please get the logs es-test2-0-90-0-beta1\_\*\*LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > from https://\*\*[GitHub - dongaihua/shares: share something](http://github.com/dongaihua/shares)[https://github.com/dongaihua/shares](https://github.com/dongaihua/shares)  
> > > Thank you.
> > > 
> > > -Regards-  
> > > -Dong aihua-
> > > 
> > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，\*\*Dong Aihua写道：
> > > 
> > > > I uploaded several times for the logs. All failed, I got the 340 error.  
> > > > I will upload the logs later.
> > > > 
> > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，\*\*Dong Aihua写道：
> > > > 
> > > > > Hi, Simon:  
> > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > indices.recovery.\*\*internal\_action\_timeout: 30s.  
> > > > > The same problem happened the again.  
> > > > > The configuration is same as before. The following are my test steps:
> > > > > 
> > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > {"ok":true,"acknowledged":\*\*true}  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 15,  
> > > > > "active\_shards" : 28,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 2,  
> > > > > "unassigned\_shards" : 0  
> > > > > }  
> > > > > The cluster stayed in this state for long time
> > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > {"ok":true,"acknowledged":\*\*true}  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 30,  
> > > > > "active\_shards" : 57,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 3,  
> > > > > "unassigned\_shards" : 0  
> > > > > }
> > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 45,  
> > > > > "active\_shards" : 87,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 3,  
> > > > > "unassigned\_shards" : 0  
> > > > > }  
> > > > > The cluster stayed in this state for long time
> > > > > 4. curl -XPUT 10.96.250.214:10200/\*\*xxxxxyyyyy111[http://10.96.250.214:10200/xxxxxyyyyy111](http://10.96.250.214:10200/xxxxxyyyyy111)  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "red",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 59,  
> > > > > "active\_shards" : 112,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 7,  
> > > > > "unassigned\_shards" : 1  
> > > > > }  
> > > > > The cluster stayed in this state for long time.  
> > > > > The detail logs of 20 nodes are attached.  
> > > > > Thank you.
> > > > > 
> > > > > -Regards-  
> > > > > -Dong Aihua-
> > > > > 
> > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，\*\*simonw写道：
> > > > > 
> > > > > > Hey,
> > > > > > 
> > > > > > thanks for testing this. I can see some exception during recovery  
> > > > > > when starting a replica which causes one of the machines to wait for an  
> > > > > > answer but it doesn't come back and it doesn't seem to get notified that  
> > > > > > the connection is closed. Can you try to set: "  
> > > > > > indices.recovery.internal\_\*\*action\_timeout: 30s"  
> > > > > > So we can see if this happens just because of the closed connections  
> > > > > > here?
> > > > > > 
> > > > > > simon
> > > > > > 
> > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > 
> > > > > > > Hi, Clint:  
> > > > > > > I already opened an issue [https://github.com/](https://github.com/)\*\*  
> > > > > > > elasticsearch/elasticsearch/\*\*issues/2714[https://github.com/elasticsearch/elasticsearch/issues/2714](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has the  
> > > > > > > same problem.  
> > > > > > > My test configure is as following:  
> > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > My test step is as following:
> > > > > > > 
> > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > {"ok":true,"acknowledged":\*\*true}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/\*\*health?pretty=true[http://10.96.250.211:10200/\_cluster/health?pretty=true](http://10.96.250.211:10200/_cluster/health?pretty=true)  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "yellow",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > "active\_shards" : 27,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 3,  
> > > > > > > "unassigned\_shards" : 0  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time.
> > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > {"ok":true,"acknowledged":\*\*false}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/\*\*health?pretty=true[http://10.96.250.211:10200/\_cluster/health?pretty=true](http://10.96.250.211:10200/_cluster/health?pretty=true)  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "red",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > "active\_shards" : 53,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 6,  
> > > > > > > "unassigned\_shards" : 1  
> > > > > > > The cluster stayed in this state for long time
> > > > > > > 3. Then I created one more empty index:  
> > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > {"ok":true,"acknowledged":\*\*false}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/\*\*health?pretty=true[http://10.96.250.211:10200/\_cluster/health?pretty=true](http://10.96.250.211:10200/_cluster/health?pretty=true)  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "red",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > "active\_shards" : 78,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 6,  
> > > > > > > "unassigned\_shards" : 6  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time.  
> > > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > > Thank you.
> > > > > > > 
> > > > > > > -Regards-  
> > > > > > > -Dong Aihua-
> > > > > > > 
> > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，\*\*Clinton Gormley写道：
> > > > > > > 
> > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > 
> > > > > > > > > Hi, guys:  
> > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > 0.20.5  
> > > > > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > > > > state.  
> > > > > > > > > The following are my test steps:
> > > > > > > > > 
> > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up.  
> > > > > > > > > 3  
> > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are  
> > > > > > > > > data  
> > > > > > > > > nodes
> > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > indices for  
> > > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > > 
> > > > > > > > Please can you open an issue on github, with a full recreation of  
> > > > > > > > what  
> > > > > > > > you need to do to recreate this problem, plus all the logs from all  
> > > > > > > > of  
> > > > > > > > the nodes.
> > > > > > > > 
> > > > > > > > ta
> > > > > > > > 
> > > > > > > > Clint
> > > > > > > > 
> > > > > > > > --  
> > > > > > > > You received this message because you are subscribed to the Google Groups  
> > > > > > > > "elasticsearch" group.  
> > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > > > > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 8, 2013, 5:32am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/11 "2013-03-08T05:32:39Z")

</div>

Hi, Simon:  
The problem is if I switched to 0.19.11 version and with the same  
environment, same test steps, the cluster works fine.  
In fact, we have two test clusters(one is this which have 20 nodes,  
another cluster has 25 nodes). Both clusters have the same problem. If  
using 0.19.11 version, both clusters work fine. But if using 0.20.4, 0.20.5  
or 0.90.0.Beta1, both clusters have the same problem.  
I also tried to reproduce the problem with 2 nodes with several  
instances, the problem didn't happen.  
My test step is very simple. Just setup up the cluster and try to create  
1~3 empty indices.  
In fact, this problem caused our system crashed after we upgraded es from  
0.19.11 to 0.20.4 and all the data are lost.  
Thank you for your response.  
@other people, By the way, did others setup up the large cluster with  
0.20.4, 0.20.5 or 0.90.0.Beta1? Can you share your result?

-Regards-  
-Dong Aihua-

在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：

> So, I looked closer at the latest logs and I see a lot of disconnects  
> going on giving me the impression you have some network issues.  
> Nevertheless we pushed some stuff to detect these situations earlier but  
> non of us was able to reproduce your issues. the only thing I can ask you  
> for is to try again with latest master to see if those commits helped in  
> any way?  
> What is your setup by the way, any idea why servers disconnect all the  
> time?
> 
> simon
> 
> On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> 
> > Hi, :  
> > Is there any clue for this problem's root cause?  
> > Thank you.
> > 
> > -Regards-  
> > -Dong Aihua-
> > 
> > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > 
> > > Hi, Simon  
> > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > Thank you.
> > > 
> > > -Regards-  
> > > -Dong aihua-
> > > 
> > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > 
> > > > I uploaded several times for the logs. All failed, I got the 340 error.  
> > > > I will upload the logs later.
> > > > 
> > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > 
> > > > > Hi, Simon:  
> > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > The same problem happened the again.  
> > > > > The configuration is same as before. The following are my test steps:
> > > > > 
> > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > {"ok":true,"acknowledged":true}  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 15,  
> > > > > "active\_shards" : 28,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 2,  
> > > > > "unassigned\_shards" : 0  
> > > > > }  
> > > > > The cluster stayed in this state for long time
> > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > {"ok":true,"acknowledged":true}  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 30,  
> > > > > "active\_shards" : 57,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 3,  
> > > > > "unassigned\_shards" : 0  
> > > > > }
> > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 45,  
> > > > > "active\_shards" : 87,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 3,  
> > > > > "unassigned\_shards" : 0  
> > > > > }  
> > > > > The cluster stayed in this state for long time
> > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "red",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 59,  
> > > > > "active\_shards" : 112,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 7,  
> > > > > "unassigned\_shards" : 1  
> > > > > }  
> > > > > The cluster stayed in this state for long time.  
> > > > > The detail logs of 20 nodes are attached.  
> > > > > Thank you.
> > > > > 
> > > > > -Regards-  
> > > > > -Dong Aihua-
> > > > > 
> > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > 
> > > > > > Hey,
> > > > > > 
> > > > > > thanks for testing this. I can see some exception during recovery  
> > > > > > when starting a replica which causes one of the machines to wait for an  
> > > > > > answer but it doesn't come back and it doesn't seem to get notified that  
> > > > > > the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > 30s"  
> > > > > > So we can see if this happens just because of the closed connections  
> > > > > > here?
> > > > > > 
> > > > > > simon
> > > > > > 
> > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > 
> > > > > > > Hi, Clint:  
> > > > > > > I already opened an issue  
> > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has the  
> > > > > > > same problem.  
> > > > > > > My test configure is as following:  
> > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > My test step is as following:
> > > > > > > 
> > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "yellow",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > "active\_shards" : 27,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 3,  
> > > > > > > "unassigned\_shards" : 0  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time.
> > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "red",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > "active\_shards" : 53,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 6,  
> > > > > > > "unassigned\_shards" : 1  
> > > > > > > The cluster stayed in this state for long time
> > > > > > > 3. Then I created one more empty index:  
> > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "red",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > "active\_shards" : 78,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 6,  
> > > > > > > "unassigned\_shards" : 6  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time.  
> > > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > > Thank you.
> > > > > > > 
> > > > > > > -Regards-  
> > > > > > > -Dong Aihua-
> > > > > > > 
> > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > 
> > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > 
> > > > > > > > > Hi, guys:  
> > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > 0.20.5  
> > > > > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > > > > state.  
> > > > > > > > > The following are my test steps:
> > > > > > > > > 
> > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up.  
> > > > > > > > > 3  
> > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are  
> > > > > > > > > data  
> > > > > > > > > nodes
> > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > indices for  
> > > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > > 
> > > > > > > > Please can you open an issue on github, with a full recreation of  
> > > > > > > > what  
> > > > > > > > you need to do to recreate this problem, plus all the logs from all  
> > > > > > > > of  
> > > > > > > > the nodes.
> > > > > > > > 
> > > > > > > > ta
> > > > > > > > 
> > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 11, 2013, 2:14am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/12 "2013-03-11T02:14:01Z")

</div>

By the way, for the servers disconnect at the end of logs, the reason is I  
did a shutdown command after the test finish

在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：

> So, I looked closer at the latest logs and I see a lot of disconnects  
> going on giving me the impression you have some network issues.  
> Nevertheless we pushed some stuff to detect these situations earlier but  
> non of us was able to reproduce your issues. the only thing I can ask you  
> for is to try again with latest master to see if those commits helped in  
> any way?  
> What is your setup by the way, any idea why servers disconnect all the  
> time?
> 
> simon
> 
> On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> 
> > Hi, :  
> > Is there any clue for this problem's root cause?  
> > Thank you.
> > 
> > -Regards-  
> > -Dong Aihua-
> > 
> > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > 
> > > Hi, Simon  
> > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > Thank you.
> > > 
> > > -Regards-  
> > > -Dong aihua-
> > > 
> > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > 
> > > > I uploaded several times for the logs. All failed, I got the 340 error.  
> > > > I will upload the logs later.
> > > > 
> > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > 
> > > > > Hi, Simon:  
> > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > The same problem happened the again.  
> > > > > The configuration is same as before. The following are my test steps:
> > > > > 
> > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > {"ok":true,"acknowledged":true}  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 15,  
> > > > > "active\_shards" : 28,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 2,  
> > > > > "unassigned\_shards" : 0  
> > > > > }  
> > > > > The cluster stayed in this state for long time
> > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > {"ok":true,"acknowledged":true}  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 30,  
> > > > > "active\_shards" : 57,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 3,  
> > > > > "unassigned\_shards" : 0  
> > > > > }
> > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "yellow",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 45,  
> > > > > "active\_shards" : 87,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 3,  
> > > > > "unassigned\_shards" : 0  
> > > > > }  
> > > > > The cluster stayed in this state for long time
> > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > {  
> > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > "status" : "red",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 59,  
> > > > > "active\_shards" : 112,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 7,  
> > > > > "unassigned\_shards" : 1  
> > > > > }  
> > > > > The cluster stayed in this state for long time.  
> > > > > The detail logs of 20 nodes are attached.  
> > > > > Thank you.
> > > > > 
> > > > > -Regards-  
> > > > > -Dong Aihua-
> > > > > 
> > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > 
> > > > > > Hey,
> > > > > > 
> > > > > > thanks for testing this. I can see some exception during recovery  
> > > > > > when starting a replica which causes one of the machines to wait for an  
> > > > > > answer but it doesn't come back and it doesn't seem to get notified that  
> > > > > > the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > 30s"  
> > > > > > So we can see if this happens just because of the closed connections  
> > > > > > here?
> > > > > > 
> > > > > > simon
> > > > > > 
> > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > 
> > > > > > > Hi, Clint:  
> > > > > > > I already opened an issue  
> > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has the  
> > > > > > > same problem.  
> > > > > > > My test configure is as following:  
> > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > My test step is as following:
> > > > > > > 
> > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "yellow",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > "active\_shards" : 27,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 3,  
> > > > > > > "unassigned\_shards" : 0  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time.
> > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "red",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > "active\_shards" : 53,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 6,  
> > > > > > > "unassigned\_shards" : 1  
> > > > > > > The cluster stayed in this state for long time
> > > > > > > 3. Then I created one more empty index:  
> > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > "status" : "red",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > "active\_shards" : 78,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 6,  
> > > > > > > "unassigned\_shards" : 6  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time.  
> > > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > > Thank you.
> > > > > > > 
> > > > > > > -Regards-  
> > > > > > > -Dong Aihua-
> > > > > > > 
> > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > 
> > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > 
> > > > > > > > > Hi, guys:  
> > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > 0.20.5  
> > > > > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > > > > state.  
> > > > > > > > > The following are my test steps:
> > > > > > > > > 
> > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up.  
> > > > > > > > > 3  
> > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are  
> > > > > > > > > data  
> > > > > > > > > nodes
> > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > indices for  
> > > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > > 
> > > > > > > > Please can you open an issue on github, with a full recreation of  
> > > > > > > > what  
> > > > > > > > you need to do to recreate this problem, plus all the logs from all  
> > > > > > > > of  
> > > > > > > > the nodes.
> > > > > > > > 
> > > > > > > > ta
> > > > > > > > 
> > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\
**Post date:** [March 11, 2013, 7:54am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/13 "2013-03-11T07:54:16Z")

</div>

On Thursday, March 7, 2013 10:38:30 PM UTC+1, Ivan Brusic wrote:

> Hi Simon,
> 
> Which commits were they? 9a25867bfe154357165c87a7b509029ff832efa4? Curious  
> to see what has changed.

yeah that is what I referred to:

> <https://github.com/elastic/elasticsearch/commit/9a25867bfe154357165c87a7b509029ff832efa4>
>
> fixes #2733

simon

> I have not looked at Dong's logs, but we have also experienced nodes being  
> removed from a cluster although the process is running. One possible  
> culprit is unresponsiveness due to GC.
> 
> --  
> Ivan
> 
> On Thu, Mar 7, 2013 at 3:04 AM, simonw \<[simon.w...@elasticsearch.com](mailto:simon.w...@elasticsearch.com)\<javascript:\>
> 
> > wrote:
> 
> > So, I looked closer at the latest logs and I see a lot of disconnects  
> > going on giving me the impression you have some network issues.  
> > Nevertheless we pushed some stuff to detect these situations earlier but  
> > non of us was able to reproduce your issues. the only thing I can ask you  
> > for is to try again with latest master to see if those commits helped in  
> > any way?  
> > What is your setup by the way, any idea why servers disconnect all the  
> > time?
> > 
> > simon
> > 
> > On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> > 
> > > Hi, :  
> > > Is there any clue for this problem's root cause?  
> > > Thank you.
> > > 
> > > -Regards-  
> > > -Dong Aihua-
> > > 
> > > 在 2013年3月5日星期二UTC+8下午2时42分55秒，\*\*Dong Aihua写道：
> > > 
> > > > Hi, Simon  
> > > > Please get the logs es-test2-0-90-0-beta1\_\*\*LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > > from https://\*\*[GitHub - dongaihua/shares: share something](http://github.com/dongaihua/shares)[https://github.com/dongaihua/shares](https://github.com/dongaihua/shares)  
> > > > Thank you.
> > > > 
> > > > -Regards-  
> > > > -Dong aihua-
> > > > 
> > > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，\*\*Dong Aihua写道：
> > > > 
> > > > > I uploaded several times for the logs. All failed, I got the 340  
> > > > > error. I will upload the logs later.
> > > > > 
> > > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，\*\*Dong Aihua写道：
> > > > > 
> > > > > > Hi, Simon:  
> > > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > > indices.recovery.\*\*internal\_action\_timeout: 30s.  
> > > > > > The same problem happened the again.  
> > > > > > The configuration is same as before. The following are my test  
> > > > > > steps:
> > > > > > 
> > > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > > {"ok":true,"acknowledged":\*\*true}  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > "status" : "yellow",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 15,  
> > > > > > "active\_shards" : 28,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 2,  
> > > > > > "unassigned\_shards" : 0  
> > > > > > }  
> > > > > > The cluster stayed in this state for long time
> > > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > {"ok":true,"acknowledged":\*\*true}  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > "status" : "yellow",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 30,  
> > > > > > "active\_shards" : 57,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 3,  
> > > > > > "unassigned\_shards" : 0  
> > > > > > }
> > > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > "status" : "yellow",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 45,  
> > > > > > "active\_shards" : 87,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 3,  
> > > > > > "unassigned\_shards" : 0  
> > > > > > }  
> > > > > > The cluster stayed in this state for long time
> > > > > > 4. curl -XPUT 10.96.250.214:10200/\*\*xxxxxyyyyy111[http://10.96.250.214:10200/xxxxxyyyyy111](http://10.96.250.214:10200/xxxxxyyyyy111)  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > "status" : "red",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 59,  
> > > > > > "active\_shards" : 112,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 7,  
> > > > > > "unassigned\_shards" : 1  
> > > > > > }  
> > > > > > The cluster stayed in this state for long time.  
> > > > > > The detail logs of 20 nodes are attached.  
> > > > > > Thank you.
> > > > > > 
> > > > > > -Regards-  
> > > > > > -Dong Aihua-
> > > > > > 
> > > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，\*\*simonw写道：
> > > > > > 
> > > > > > > Hey,
> > > > > > > 
> > > > > > > thanks for testing this. I can see some exception during recovery  
> > > > > > > when starting a replica which causes one of the machines to wait for an  
> > > > > > > answer but it doesn't come back and it doesn't seem to get notified that  
> > > > > > > the connection is closed. Can you try to set: "  
> > > > > > > indices.recovery.internal\_\*\*action\_timeout: 30s"  
> > > > > > > So we can see if this happens just because of the closed connections  
> > > > > > > here?
> > > > > > > 
> > > > > > > simon
> > > > > > > 
> > > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > > 
> > > > > > > > Hi, Clint:  
> > > > > > > > I already opened an issue [https://github.com/](https://github.com/)\*\*  
> > > > > > > > elasticsearch/elasticsearch/\*\*issues/2714[https://github.com/elasticsearch/elasticsearch/issues/2714](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has  
> > > > > > > > the same problem.  
> > > > > > > > My test configure is as following:  
> > > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > > My test step is as following:
> > > > > > > > 
> > > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > {"ok":true,"acknowledged":\*\*true}  
> > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/\*\*health?pretty=true[http://10.96.250.211:10200/\_cluster/health?pretty=true](http://10.96.250.211:10200/_cluster/health?pretty=true)  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > "status" : "yellow",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > "active\_shards" : 27,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > }  
> > > > > > > > The cluster stayed in this state for long time.
> > > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > {"ok":true,"acknowledged":\*\*false}  
> > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/\*\*health?pretty=true[http://10.96.250.211:10200/\_cluster/health?pretty=true](http://10.96.250.211:10200/_cluster/health?pretty=true)  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > "status" : "red",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > > "active\_shards" : 53,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > The cluster stayed in this state for long time
> > > > > > > > 3. Then I created one more empty index:  
> > > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > {"ok":true,"acknowledged":\*\*false}  
> > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/\*\*health?pretty=true[http://10.96.250.211:10200/\_cluster/health?pretty=true](http://10.96.250.211:10200/_cluster/health?pretty=true)  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > "status" : "red",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > > "active\_shards" : 78,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > "unassigned\_shards" : 6  
> > > > > > > > }  
> > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > > > Thank you.
> > > > > > > > 
> > > > > > > > -Regards-  
> > > > > > > > -Dong Aihua-
> > > > > > > > 
> > > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，\*\*Clinton Gormley写道：
> > > > > > > > 
> > > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > > 
> > > > > > > > > > Hi, guys:  
> > > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > > 0.20.5  
> > > > > > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > > > > > state.  
> > > > > > > > > > The following are my test steps:
> > > > > > > > > > 
> > > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up.  
> > > > > > > > > > 3  
> > > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are  
> > > > > > > > > > data  
> > > > > > > > > > nodes
> > > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > > indices for  
> > > > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > > > 
> > > > > > > > > Please can you open an issue on github, with a full recreation of  
> > > > > > > > > what  
> > > > > > > > > you need to do to recreate this problem, plus all the logs from  
> > > > > > > > > all of  
> > > > > > > > > the nodes.
> > > > > > > > > 
> > > > > > > > > ta
> > > > > > > > > 
> > > > > > > > > Clint
> > > > > > > > > 
> > > > > > > > > --  
> > > > > > > > > You received this message because you are subscribed to the Google Groups  
> > > > > > > > > "elasticsearch" group.  
> > > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > > > > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > > > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\
**Post date:** [March 11, 2013, 7:56am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/14 "2013-03-11T07:56:07Z")

</div>

Thanks for the headsup, that is what I figured. I am just wondering what  
triggered the intermediate disconnects that causes the servers to wait on  
their peers they recover from. Can you tell a little about your setup, are  
they VMs do you have firewalls in place. Are the servers in different  
datascenters, what is the connection between them?

simon

On Monday, March 11, 2013 3:14:01 AM UTC+1, Dong Aihua wrote:

> By the way, for the servers disconnect at the end of logs, the reason is I  
> did a shutdown command after the test finish
> 
> 在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：
> 
> > So, I looked closer at the latest logs and I see a lot of disconnects  
> > going on giving me the impression you have some network issues.  
> > Nevertheless we pushed some stuff to detect these situations earlier but  
> > non of us was able to reproduce your issues. the only thing I can ask you  
> > for is to try again with latest master to see if those commits helped in  
> > any way?  
> > What is your setup by the way, any idea why servers disconnect all the  
> > time?
> > 
> > simon
> > 
> > On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> > 
> > > Hi, :  
> > > Is there any clue for this problem's root cause?  
> > > Thank you.
> > > 
> > > -Regards-  
> > > -Dong Aihua-
> > > 
> > > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > > 
> > > > Hi, Simon  
> > > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > > Thank you.
> > > > 
> > > > -Regards-  
> > > > -Dong aihua-
> > > > 
> > > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > > 
> > > > > I uploaded several times for the logs. All failed, I got the 340  
> > > > > error. I will upload the logs later.
> > > > > 
> > > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > > 
> > > > > > Hi, Simon:  
> > > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > > The same problem happened the again.  
> > > > > > The configuration is same as before. The following are my test  
> > > > > > steps:
> > > > > > 
> > > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > > {"ok":true,"acknowledged":true}  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > "status" : "yellow",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 15,  
> > > > > > "active\_shards" : 28,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 2,  
> > > > > > "unassigned\_shards" : 0  
> > > > > > }  
> > > > > > The cluster stayed in this state for long time
> > > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > {"ok":true,"acknowledged":true}  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > "status" : "yellow",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 30,  
> > > > > > "active\_shards" : 57,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 3,  
> > > > > > "unassigned\_shards" : 0  
> > > > > > }
> > > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > "status" : "yellow",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 45,  
> > > > > > "active\_shards" : 87,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 3,  
> > > > > > "unassigned\_shards" : 0  
> > > > > > }  
> > > > > > The cluster stayed in this state for long time
> > > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > > {  
> > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > "status" : "red",  
> > > > > > "timed\_out" : false,  
> > > > > > "number\_of\_nodes" : 20,  
> > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > "active\_primary\_shards" : 59,  
> > > > > > "active\_shards" : 112,  
> > > > > > "relocating\_shards" : 0,  
> > > > > > "initializing\_shards" : 7,  
> > > > > > "unassigned\_shards" : 1  
> > > > > > }  
> > > > > > The cluster stayed in this state for long time.  
> > > > > > The detail logs of 20 nodes are attached.  
> > > > > > Thank you.
> > > > > > 
> > > > > > -Regards-  
> > > > > > -Dong Aihua-
> > > > > > 
> > > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > > 
> > > > > > > Hey,
> > > > > > > 
> > > > > > > thanks for testing this. I can see some exception during recovery  
> > > > > > > when starting a replica which causes one of the machines to wait for an  
> > > > > > > answer but it doesn't come back and it doesn't seem to get notified that  
> > > > > > > the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > > 30s"  
> > > > > > > So we can see if this happens just because of the closed connections  
> > > > > > > here?
> > > > > > > 
> > > > > > > simon
> > > > > > > 
> > > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > > 
> > > > > > > > Hi, Clint:  
> > > > > > > > I already opened an issue  
> > > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has  
> > > > > > > > the same problem.  
> > > > > > > > My test configure is as following:  
> > > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > > My test step is as following:
> > > > > > > > 
> > > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > "status" : "yellow",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > "active\_shards" : 27,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > }  
> > > > > > > > The cluster stayed in this state for long time.
> > > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > "status" : "red",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > > "active\_shards" : 53,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > The cluster stayed in this state for long time
> > > > > > > > 3. Then I created one more empty index:  
> > > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > "status" : "red",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > > "active\_shards" : 78,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > "unassigned\_shards" : 6  
> > > > > > > > }  
> > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > > > Thank you.
> > > > > > > > 
> > > > > > > > -Regards-  
> > > > > > > > -Dong Aihua-
> > > > > > > > 
> > > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > > 
> > > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > > 
> > > > > > > > > > Hi, guys:  
> > > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > > 0.20.5  
> > > > > > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > > > > > state.  
> > > > > > > > > > The following are my test steps:
> > > > > > > > > > 
> > > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster up.  
> > > > > > > > > > 3  
> > > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are  
> > > > > > > > > > data  
> > > > > > > > > > nodes
> > > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > > indices for  
> > > > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > > > 
> > > > > > > > > Please can you open an issue on github, with a full recreation of  
> > > > > > > > > what  
> > > > > > > > > you need to do to recreate this problem, plus all the logs from  
> > > > > > > > > all of  
> > > > > > > > > the nodes.
> > > > > > > > > 
> > > > > > > > > ta
> > > > > > > > > 
> > > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 11, 2013, 10:10am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/15 "2013-03-11T10:10:08Z")

</div>

Hi, Simon:  
Today I tested 0.19.12, it is fine.  
Then I tested 0.20.0, the problem happened again. So I guess the problem  
is introduced since 0.20.0.  
The test steps are as before. And the following are some logs:  
{  
"cluster\_name" : "test-0.20.0",  
"status" : "red",  
"timed\_out" : false,  
"number\_of\_nodes" : 20,  
"number\_of\_data\_nodes" : 15,  
"active\_primary\_shards" : 224,  
"active\_shards" : 445,  
"relocating\_shards" : 0,  
"initializing\_shards" : 4,  
"unassigned\_shards" : 1  
}  
{ cluster\_name: 'test-0.20.0',  
master\_node:  
{ name: 'xseed021.kdev',  
transport\_address: 'inet[/10.96.250.211:9400]',  
attributes: { data: 'false', master: 'true' } },  
initShards:  
[ { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
primary: false,  
shard: 5,  
index: 'testx-xxx-2012-03-11' },  
{ node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
primary: false,  
shard: 11,  
index: 'test-2013-03-11' },  
{ node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
primary: true,  
shard: 7,  
index: 'testx-xxx-2013-zzz' },  
{ node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
primary: false,  
shard: 5,  
index: 'testx-xxx-2013-yyy' } ],  
unassigned\_shards\_total: 1,  
unassigned\_indices: ['testx-xxx-2013-zzz'] }  
The detail logs are located at [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares/)  
test-0.20.0\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/test-0.20.0\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/test-0.20.0_LOGS.tar.gz)  
By the way, the servers are setup by IT team, I'm not clear the physical  
connections. If you really need those informaiton, I can ask and give you  
response.  
Thank you.

-Regards-  
-dongaihua-

在 2013年3月11日星期一UTC+8下午3时56分07秒，simonw写道：

> Thanks for the headsup, that is what I figured. I am just wondering what  
> triggered the intermediate disconnects that causes the servers to wait on  
> their peers they recover from. Can you tell a little about your setup, are  
> they VMs do you have firewalls in place. Are the servers in different  
> datascenters, what is the connection between them?
> 
> simon
> 
> On Monday, March 11, 2013 3:14:01 AM UTC+1, Dong Aihua wrote:
> 
> > By the way, for the servers disconnect at the end of logs, the reason is  
> > I did a shutdown command after the test finish
> > 
> > 在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：
> > 
> > > So, I looked closer at the latest logs and I see a lot of disconnects  
> > > going on giving me the impression you have some network issues.  
> > > Nevertheless we pushed some stuff to detect these situations earlier but  
> > > non of us was able to reproduce your issues. the only thing I can ask you  
> > > for is to try again with latest master to see if those commits helped in  
> > > any way?  
> > > What is your setup by the way, any idea why servers disconnect all the  
> > > time?
> > > 
> > > simon
> > > 
> > > On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> > > 
> > > > Hi, :  
> > > > Is there any clue for this problem's root cause?  
> > > > Thank you.
> > > > 
> > > > -Regards-  
> > > > -Dong Aihua-
> > > > 
> > > > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > > > 
> > > > > Hi, Simon  
> > > > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > > > Thank you.
> > > > > 
> > > > > -Regards-  
> > > > > -Dong aihua-
> > > > > 
> > > > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > > > 
> > > > > > I uploaded several times for the logs. All failed, I got the 340  
> > > > > > error. I will upload the logs later.
> > > > > > 
> > > > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > > > 
> > > > > > > Hi, Simon:  
> > > > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > > > The same problem happened the again.  
> > > > > > > The configuration is same as before. The following are my test  
> > > > > > > steps:
> > > > > > > 
> > > > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > "status" : "yellow",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > "active\_shards" : 28,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 2,  
> > > > > > > "unassigned\_shards" : 0  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time
> > > > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > "status" : "yellow",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 30,  
> > > > > > > "active\_shards" : 57,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 3,  
> > > > > > > "unassigned\_shards" : 0  
> > > > > > > }
> > > > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > "status" : "yellow",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 45,  
> > > > > > > "active\_shards" : 87,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 3,  
> > > > > > > "unassigned\_shards" : 0  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time
> > > > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > > > {  
> > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > "status" : "red",  
> > > > > > > "timed\_out" : false,  
> > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > "active\_primary\_shards" : 59,  
> > > > > > > "active\_shards" : 112,  
> > > > > > > "relocating\_shards" : 0,  
> > > > > > > "initializing\_shards" : 7,  
> > > > > > > "unassigned\_shards" : 1  
> > > > > > > }  
> > > > > > > The cluster stayed in this state for long time.  
> > > > > > > The detail logs of 20 nodes are attached.  
> > > > > > > Thank you.
> > > > > > > 
> > > > > > > -Regards-  
> > > > > > > -Dong Aihua-
> > > > > > > 
> > > > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > > > 
> > > > > > > > Hey,
> > > > > > > > 
> > > > > > > > thanks for testing this. I can see some exception during recovery  
> > > > > > > > when starting a replica which causes one of the machines to wait for an  
> > > > > > > > answer but it doesn't come back and it doesn't seem to get notified that  
> > > > > > > > the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > > > 30s"  
> > > > > > > > So we can see if this happens just because of the closed  
> > > > > > > > connections here?
> > > > > > > > 
> > > > > > > > simon
> > > > > > > > 
> > > > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > > > 
> > > > > > > > > Hi, Clint:  
> > > > > > > > > I already opened an issue  
> > > > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has  
> > > > > > > > > the same problem.  
> > > > > > > > > My test configure is as following:  
> > > > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > > > My test step is as following:
> > > > > > > > > 
> > > > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > {  
> > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > "status" : "yellow",  
> > > > > > > > > "timed\_out" : false,  
> > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > "active\_shards" : 27,  
> > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > }  
> > > > > > > > > The cluster stayed in this state for long time.
> > > > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > {  
> > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > "status" : "red",  
> > > > > > > > > "timed\_out" : false,  
> > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > > > "active\_shards" : 53,  
> > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > 3. Then I created one more empty index:  
> > > > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > {  
> > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > "status" : "red",  
> > > > > > > > > "timed\_out" : false,  
> > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > > > "active\_shards" : 78,  
> > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > "unassigned\_shards" : 6  
> > > > > > > > > }  
> > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > > > > Thank you.
> > > > > > > > > 
> > > > > > > > > -Regards-  
> > > > > > > > > -Dong Aihua-
> > > > > > > > > 
> > > > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > > > 
> > > > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi, guys:  
> > > > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > > > 0.20.5  
> > > > > > > > > > > which cause shards allocation failure and stuck in initializing  
> > > > > > > > > > > state.  
> > > > > > > > > > > The following are my test steps:
> > > > > > > > > > > 
> > > > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster  
> > > > > > > > > > > up. 3  
> > > > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes are  
> > > > > > > > > > > data  
> > > > > > > > > > > nodes
> > > > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > > > indices for  
> > > > > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > > > > 
> > > > > > > > > > Please can you open an issue on github, with a full recreation of  
> > > > > > > > > > what  
> > > > > > > > > > you need to do to recreate this problem, plus all the logs from  
> > > > > > > > > > all of  
> > > > > > > > > > the nodes.
> > > > > > > > > > 
> > > > > > > > > > ta
> > > > > > > > > > 
> > > > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 28, 2013, 2:44am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/16 "2013-03-28T02:44:10Z")

</div>

Hi, Simon:  
Today I tested the latest ES version 0.20.6, the problem seems fixed in  
this version.  
Can you confirm it? And I'm wondering how do you fix it?  
Thanks a lot.

-Regards-  
-dongaihua-

在 2013年3月11日星期一UTC+8下午6时10分08秒，Dong Aihua写道：

> Hi, Simon:  
> Today I tested 0.19.12, it is fine.  
> Then I tested 0.20.0, the problem happened again. So I guess the problem  
> is introduced since 0.20.0.  
> The test steps are as before. And the following are some logs:  
> {  
> "cluster\_name" : "test-0.20.0",  
> "status" : "red",  
> "timed\_out" : false,  
> "number\_of\_nodes" : 20,  
> "number\_of\_data\_nodes" : 15,  
> "active\_primary\_shards" : 224,  
> "active\_shards" : 445,  
> "relocating\_shards" : 0,  
> "initializing\_shards" : 4,  
> "unassigned\_shards" : 1  
> }  
> { cluster\_name: 'test-0.20.0',  
> master\_node:  
> { name: 'xseed021.kdev',  
> transport\_address: 'inet[/10.96.250.211:9400]',  
> attributes: { data: 'false', master: 'true' } },  
> initShards:  
> [ { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> primary: false,  
> shard: 5,  
> index: 'testx-xxx-2012-03-11' },  
> { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> primary: false,  
> shard: 11,  
> index: 'test-2013-03-11' },  
> { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> primary: true,  
> shard: 7,  
> index: 'testx-xxx-2013-zzz' },  
> { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> primary: false,  
> shard: 5,  
> index: 'testx-xxx-2013-yyy' } ],  
> unassigned\_shards\_total: 1,  
> unassigned\_indices: ['testx-xxx-2013-zzz'] }  
> The detail logs are located at [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares/)  
> test-0.20.0\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/test-0.20.0\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/test-0.20.0_LOGS.tar.gz)  
> By the way, the servers are setup by IT team, I'm not clear the physical  
> connections. If you really need those informaiton, I can ask and give you  
> response.  
> Thank you.
> 
> -Regards-  
> -dongaihua-
> 
> 在 2013年3月11日星期一UTC+8下午3时56分07秒，simonw写道：
> 
> > Thanks for the headsup, that is what I figured. I am just wondering what  
> > triggered the intermediate disconnects that causes the servers to wait on  
> > their peers they recover from. Can you tell a little about your setup, are  
> > they VMs do you have firewalls in place. Are the servers in different  
> > datascenters, what is the connection between them?
> > 
> > simon
> > 
> > On Monday, March 11, 2013 3:14:01 AM UTC+1, Dong Aihua wrote:
> > 
> > > By the way, for the servers disconnect at the end of logs, the reason is  
> > > I did a shutdown command after the test finish
> > > 
> > > 在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：
> > > 
> > > > So, I looked closer at the latest logs and I see a lot of disconnects  
> > > > going on giving me the impression you have some network issues.  
> > > > Nevertheless we pushed some stuff to detect these situations earlier but  
> > > > non of us was able to reproduce your issues. the only thing I can ask you  
> > > > for is to try again with latest master to see if those commits helped in  
> > > > any way?  
> > > > What is your setup by the way, any idea why servers disconnect all the  
> > > > time?
> > > > 
> > > > simon
> > > > 
> > > > On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> > > > 
> > > > > Hi, :  
> > > > > Is there any clue for this problem's root cause?  
> > > > > Thank you.
> > > > > 
> > > > > -Regards-  
> > > > > -Dong Aihua-
> > > > > 
> > > > > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > > > > 
> > > > > > Hi, Simon  
> > > > > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > > > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > > > > Thank you.
> > > > > > 
> > > > > > -Regards-  
> > > > > > -Dong aihua-
> > > > > > 
> > > > > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > > > > 
> > > > > > > I uploaded several times for the logs. All failed, I got the 340  
> > > > > > > error. I will upload the logs later.
> > > > > > > 
> > > > > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > > > > 
> > > > > > > > Hi, Simon:  
> > > > > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > > > > The same problem happened the again.  
> > > > > > > > The configuration is same as before. The following are my test  
> > > > > > > > steps:
> > > > > > > > 
> > > > > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > "status" : "yellow",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > "active\_shards" : 28,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 2,  
> > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > }  
> > > > > > > > The cluster stayed in this state for long time
> > > > > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > "status" : "yellow",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 30,  
> > > > > > > > "active\_shards" : 57,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > }
> > > > > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > "status" : "yellow",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 45,  
> > > > > > > > "active\_shards" : 87,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > }  
> > > > > > > > The cluster stayed in this state for long time
> > > > > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > > > > {  
> > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > "status" : "red",  
> > > > > > > > "timed\_out" : false,  
> > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > "active\_primary\_shards" : 59,  
> > > > > > > > "active\_shards" : 112,  
> > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > "initializing\_shards" : 7,  
> > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > }  
> > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > The detail logs of 20 nodes are attached.  
> > > > > > > > Thank you.
> > > > > > > > 
> > > > > > > > -Regards-  
> > > > > > > > -Dong Aihua-
> > > > > > > > 
> > > > > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > > > > 
> > > > > > > > > Hey,
> > > > > > > > > 
> > > > > > > > > thanks for testing this. I can see some exception during recovery  
> > > > > > > > > when starting a replica which causes one of the machines to wait for an  
> > > > > > > > > answer but it doesn't come back and it doesn't seem to get notified that  
> > > > > > > > > the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > > > > 30s"  
> > > > > > > > > So we can see if this happens just because of the closed  
> > > > > > > > > connections here?
> > > > > > > > > 
> > > > > > > > > simon
> > > > > > > > > 
> > > > > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > > > > 
> > > > > > > > > > Hi, Clint:  
> > > > > > > > > > I already opened an issue  
> > > > > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has  
> > > > > > > > > > the same problem.  
> > > > > > > > > > My test configure is as following:  
> > > > > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > > > > My test step is as following:
> > > > > > > > > > 
> > > > > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > {  
> > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > > "active\_shards" : 27,  
> > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > }  
> > > > > > > > > > The cluster stayed in this state for long time.
> > > > > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > {  
> > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > "status" : "red",  
> > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > > > > "active\_shards" : 53,  
> > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > 3. Then I created one more empty index:  
> > > > > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > {  
> > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > "status" : "red",  
> > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > > > > "active\_shards" : 78,  
> > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > "unassigned\_shards" : 6  
> > > > > > > > > > }  
> > > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > > > > > Thank you.
> > > > > > > > > > 
> > > > > > > > > > -Regards-  
> > > > > > > > > > -Dong Aihua-
> > > > > > > > > > 
> > > > > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > > > > 
> > > > > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hi, guys:  
> > > > > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > > > > 0.20.5  
> > > > > > > > > > > > which cause shards allocation failure and stuck in  
> > > > > > > > > > > > initializing state.  
> > > > > > > > > > > > The following are my test steps:
> > > > > > > > > > > > 
> > > > > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster  
> > > > > > > > > > > > up. 3  
> > > > > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes  
> > > > > > > > > > > > are data  
> > > > > > > > > > > > nodes
> > > > > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > > > > indices for  
> > > > > > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > > > > > 
> > > > > > > > > > > Please can you open an issue on github, with a full recreation  
> > > > > > > > > > > of what  
> > > > > > > > > > > you need to do to recreate this problem, plus all the logs from  
> > > > > > > > > > > all of  
> > > > > > > > > > > the nodes.
> > > > > > > > > > > 
> > > > > > > > > > > ta
> > > > > > > > > > > 
> > > > > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\
**Post date:** [March 28, 2013, 8:07am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/17 "2013-03-28T08:07:17Z")

</div>

Hmm that is interesting... I suspect that the fix for the missed  
ChannelClosed events fixed it then though....I don't have you logs around  
anymore but did you see any TooManyOpenFiles exceptions by any chance? Can  
you confirm that the latest master has fixed this as well by running a  
master version / latest build?

simon

On Thursday, March 28, 2013 3:44:10 AM UTC+1, Dong Aihua wrote:

> Hi, Simon:  
> Today I tested the latest ES version 0.20.6, the problem seems fixed in  
> this version.  
> Can you confirm it? And I'm wondering how do you fix it?  
> Thanks a lot.
> 
> -Regards-  
> -dongaihua-
> 
> 在 2013年3月11日星期一UTC+8下午6时10分08秒，Dong Aihua写道：
> 
> > Hi, Simon:  
> > Today I tested 0.19.12, it is fine.  
> > Then I tested 0.20.0, the problem happened again. So I guess the  
> > problem is introduced since 0.20.0.  
> > The test steps are as before. And the following are some logs:  
> > {  
> > "cluster\_name" : "test-0.20.0",  
> > "status" : "red",  
> > "timed\_out" : false,  
> > "number\_of\_nodes" : 20,  
> > "number\_of\_data\_nodes" : 15,  
> > "active\_primary\_shards" : 224,  
> > "active\_shards" : 445,  
> > "relocating\_shards" : 0,  
> > "initializing\_shards" : 4,  
> > "unassigned\_shards" : 1  
> > }  
> > { cluster\_name: 'test-0.20.0',  
> > master\_node:  
> > { name: 'xseed021.kdev',  
> > transport\_address: 'inet[/10.96.250.211:9400]',  
> > attributes: { data: 'false', master: 'true' } },  
> > initShards:  
> > [ { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> > primary: false,  
> > shard: 5,  
> > index: 'testx-xxx-2012-03-11' },  
> > { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> > primary: false,  
> > shard: 11,  
> > index: 'test-2013-03-11' },  
> > { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> > primary: true,  
> > shard: 7,  
> > index: 'testx-xxx-2013-zzz' },  
> > { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> > primary: false,  
> > shard: 5,  
> > index: 'testx-xxx-2013-yyy' } ],  
> > unassigned\_shards\_total: 1,  
> > unassigned\_indices: ['testx-xxx-2013-zzz'] }  
> > The detail logs are located at [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares/)  
> > test-0.20.0\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/test-0.20.0\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/test-0.20.0_LOGS.tar.gz)  
> > By the way, the servers are setup by IT team, I'm not clear the  
> > physical connections. If you really need those informaiton, I can ask and  
> > give you response.  
> > Thank you.
> > 
> > -Regards-  
> > -dongaihua-
> > 
> > 在 2013年3月11日星期一UTC+8下午3时56分07秒，simonw写道：
> > 
> > > Thanks for the headsup, that is what I figured. I am just wondering what  
> > > triggered the intermediate disconnects that causes the servers to wait on  
> > > their peers they recover from. Can you tell a little about your setup, are  
> > > they VMs do you have firewalls in place. Are the servers in different  
> > > datascenters, what is the connection between them?
> > > 
> > > simon
> > > 
> > > On Monday, March 11, 2013 3:14:01 AM UTC+1, Dong Aihua wrote:
> > > 
> > > > By the way, for the servers disconnect at the end of logs, the reason  
> > > > is I did a shutdown command after the test finish
> > > > 
> > > > 在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：
> > > > 
> > > > > So, I looked closer at the latest logs and I see a lot of disconnects  
> > > > > going on giving me the impression you have some network issues.  
> > > > > Nevertheless we pushed some stuff to detect these situations earlier but  
> > > > > non of us was able to reproduce your issues. the only thing I can ask you  
> > > > > for is to try again with latest master to see if those commits helped in  
> > > > > any way?  
> > > > > What is your setup by the way, any idea why servers disconnect all the  
> > > > > time?
> > > > > 
> > > > > simon
> > > > > 
> > > > > On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> > > > > 
> > > > > > Hi, :  
> > > > > > Is there any clue for this problem's root cause?  
> > > > > > Thank you.
> > > > > > 
> > > > > > -Regards-  
> > > > > > -Dong Aihua-
> > > > > > 
> > > > > > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > > > > > 
> > > > > > > Hi, Simon  
> > > > > > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > > > > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > > > > > Thank you.
> > > > > > > 
> > > > > > > -Regards-  
> > > > > > > -Dong aihua-
> > > > > > > 
> > > > > > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > > > > > 
> > > > > > > > I uploaded several times for the logs. All failed, I got the 340  
> > > > > > > > error. I will upload the logs later.
> > > > > > > > 
> > > > > > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > > > > > 
> > > > > > > > > Hi, Simon:  
> > > > > > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > > > > > The same problem happened the again.  
> > > > > > > > > The configuration is same as before. The following are my test  
> > > > > > > > > steps:
> > > > > > > > > 
> > > > > > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > {  
> > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > "status" : "yellow",  
> > > > > > > > > "timed\_out" : false,  
> > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > "active\_shards" : 28,  
> > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > "initializing\_shards" : 2,  
> > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > }  
> > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > {  
> > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > "status" : "yellow",  
> > > > > > > > > "timed\_out" : false,  
> > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > "active\_primary\_shards" : 30,  
> > > > > > > > > "active\_shards" : 57,  
> > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > }
> > > > > > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > {  
> > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > "status" : "yellow",  
> > > > > > > > > "timed\_out" : false,  
> > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > "active\_primary\_shards" : 45,  
> > > > > > > > > "active\_shards" : 87,  
> > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > }  
> > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > > > > > {  
> > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > "status" : "red",  
> > > > > > > > > "timed\_out" : false,  
> > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > "active\_primary\_shards" : 59,  
> > > > > > > > > "active\_shards" : 112,  
> > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > "initializing\_shards" : 7,  
> > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > }  
> > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > The detail logs of 20 nodes are attached.  
> > > > > > > > > Thank you.
> > > > > > > > > 
> > > > > > > > > -Regards-  
> > > > > > > > > -Dong Aihua-
> > > > > > > > > 
> > > > > > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > > > > > 
> > > > > > > > > > Hey,
> > > > > > > > > > 
> > > > > > > > > > thanks for testing this. I can see some exception during recovery  
> > > > > > > > > > when starting a replica which causes one of the machines to wait for an  
> > > > > > > > > > answer but it doesn't come back and it doesn't seem to get notified that  
> > > > > > > > > > the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > > > > > 30s"  
> > > > > > > > > > So we can see if this happens just because of the closed  
> > > > > > > > > > connections here?
> > > > > > > > > > 
> > > > > > > > > > simon
> > > > > > > > > > 
> > > > > > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > > > > > 
> > > > > > > > > > > Hi, Clint:  
> > > > > > > > > > > I already opened an issue  
> > > > > > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it has  
> > > > > > > > > > > the same problem.  
> > > > > > > > > > > My test configure is as following:  
> > > > > > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > > > > > My test step is as following:
> > > > > > > > > > > 
> > > > > > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > {  
> > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > > > "active\_shards" : 27,  
> > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > }  
> > > > > > > > > > > The cluster stayed in this state for long time.
> > > > > > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > {  
> > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > > > > > "active\_shards" : 53,  
> > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > > 3. Then I created one more empty index:  
> > > > > > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > {  
> > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > > > > > "active\_shards" : 78,  
> > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > > "unassigned\_shards" : 6  
> > > > > > > > > > > }  
> > > > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > > > You can refer the detail logs from 20 nodes in the attachments.  
> > > > > > > > > > > Thank you.
> > > > > > > > > > > 
> > > > > > > > > > > -Regards-  
> > > > > > > > > > > -Dong Aihua-
> > > > > > > > > > > 
> > > > > > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > > > > > 
> > > > > > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi, guys:  
> > > > > > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > > > > > 0.20.5  
> > > > > > > > > > > > > which cause shards allocation failure and stuck in  
> > > > > > > > > > > > > initializing state.  
> > > > > > > > > > > > > The following are my test steps:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster  
> > > > > > > > > > > > > up. 3  
> > > > > > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes  
> > > > > > > > > > > > > are data  
> > > > > > > > > > > > > nodes
> > > > > > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > > > > > indices for  
> > > > > > > > > > > > > example index-2013-02-25, index-2013-02-26, index-2013-02-27,  
> > > > > > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > > > > > But some shards stuck in initializing status for long time.
> > > > > > > > > > > > 
> > > > > > > > > > > > Please can you open an issue on github, with a full recreation  
> > > > > > > > > > > > of what  
> > > > > > > > > > > > you need to do to recreate this problem, plus all the logs from  
> > > > > > > > > > > > all of  
> > > > > > > > > > > > the nodes.
> > > > > > > > > > > > 
> > > > > > > > > > > > ta
> > > > > > > > > > > > 
> > > > > > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [March 28, 2013, 8:36am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/18 "2013-03-28T08:36:00Z")

</div>

Ok, I will test it and tell you the result and logs.

在 2013年3月28日星期四UTC+8下午4时07分17秒，simonw写道：

> Hmm that is interesting... I suspect that the fix for the missed  
> ChannelClosed events fixed it then though....I don't have you logs around  
> anymore but did you see any TooManyOpenFiles exceptions by any chance? Can  
> you confirm that the latest master has fixed this as well by running a  
> master version / latest build?
> 
> simon
> 
> On Thursday, March 28, 2013 3:44:10 AM UTC+1, Dong Aihua wrote:
> 
> > Hi, Simon:  
> > Today I tested the latest ES version 0.20.6, the problem seems fixed in  
> > this version.  
> > Can you confirm it? And I'm wondering how do you fix it?  
> > Thanks a lot.
> > 
> > -Regards-  
> > -dongaihua-
> > 
> > 在 2013年3月11日星期一UTC+8下午6时10分08秒，Dong Aihua写道：
> > 
> > > Hi, Simon:  
> > > Today I tested 0.19.12, it is fine.  
> > > Then I tested 0.20.0, the problem happened again. So I guess the  
> > > problem is introduced since 0.20.0.  
> > > The test steps are as before. And the following are some logs:  
> > > {  
> > > "cluster\_name" : "test-0.20.0",  
> > > "status" : "red",  
> > > "timed\_out" : false,  
> > > "number\_of\_nodes" : 20,  
> > > "number\_of\_data\_nodes" : 15,  
> > > "active\_primary\_shards" : 224,  
> > > "active\_shards" : 445,  
> > > "relocating\_shards" : 0,  
> > > "initializing\_shards" : 4,  
> > > "unassigned\_shards" : 1  
> > > }  
> > > { cluster\_name: 'test-0.20.0',  
> > > master\_node:  
> > > { name: 'xseed021.kdev',  
> > > transport\_address: 'inet[/10.96.250.211:9400]',  
> > > attributes: { data: 'false', master: 'true' } },  
> > > initShards:  
> > > [ { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> > > primary: false,  
> > > shard: 5,  
> > > index: 'testx-xxx-2012-03-11' },  
> > > { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> > > primary: false,  
> > > shard: 11,  
> > > index: 'test-2013-03-11' },  
> > > { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> > > primary: true,  
> > > shard: 7,  
> > > index: 'testx-xxx-2013-zzz' },  
> > > { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> > > primary: false,  
> > > shard: 5,  
> > > index: 'testx-xxx-2013-yyy' } ],  
> > > unassigned\_shards\_total: 1,  
> > > unassigned\_indices: ['testx-xxx-2013-zzz'] }  
> > > The detail logs are located at [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares/)  
> > > test-0.20.0\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/test-0.20.0\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/test-0.20.0_LOGS.tar.gz)  
> > > By the way, the servers are setup by IT team, I'm not clear the  
> > > physical connections. If you really need those informaiton, I can ask and  
> > > give you response.  
> > > Thank you.
> > > 
> > > -Regards-  
> > > -dongaihua-
> > > 
> > > 在 2013年3月11日星期一UTC+8下午3时56分07秒，simonw写道：
> > > 
> > > > Thanks for the headsup, that is what I figured. I am just wondering  
> > > > what triggered the intermediate disconnects that causes the servers to wait  
> > > > on their peers they recover from. Can you tell a little about your setup,  
> > > > are they VMs do you have firewalls in place. Are the servers in different  
> > > > datascenters, what is the connection between them?
> > > > 
> > > > simon
> > > > 
> > > > On Monday, March 11, 2013 3:14:01 AM UTC+1, Dong Aihua wrote:
> > > > 
> > > > > By the way, for the servers disconnect at the end of logs, the reason  
> > > > > is I did a shutdown command after the test finish
> > > > > 
> > > > > 在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：
> > > > > 
> > > > > > So, I looked closer at the latest logs and I see a lot of disconnects  
> > > > > > going on giving me the impression you have some network issues.  
> > > > > > Nevertheless we pushed some stuff to detect these situations earlier but  
> > > > > > non of us was able to reproduce your issues. the only thing I can ask you  
> > > > > > for is to try again with latest master to see if those commits helped in  
> > > > > > any way?  
> > > > > > What is your setup by the way, any idea why servers disconnect all  
> > > > > > the time?
> > > > > > 
> > > > > > simon
> > > > > > 
> > > > > > On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> > > > > > 
> > > > > > > Hi, :  
> > > > > > > Is there any clue for this problem's root cause?  
> > > > > > > Thank you.
> > > > > > > 
> > > > > > > -Regards-  
> > > > > > > -Dong Aihua-
> > > > > > > 
> > > > > > > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > > > > > > 
> > > > > > > > Hi, Simon  
> > > > > > > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > > > > > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > > > > > > Thank you.
> > > > > > > > 
> > > > > > > > -Regards-  
> > > > > > > > -Dong aihua-
> > > > > > > > 
> > > > > > > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > > > > > > 
> > > > > > > > > I uploaded several times for the logs. All failed, I got the 340  
> > > > > > > > > error. I will upload the logs later.
> > > > > > > > > 
> > > > > > > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > > > > > > 
> > > > > > > > > > Hi, Simon:  
> > > > > > > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > > > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > > > > > > The same problem happened the again.  
> > > > > > > > > > The configuration is same as before. The following are my test  
> > > > > > > > > > steps:
> > > > > > > > > > 
> > > > > > > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > {  
> > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > > "active\_shards" : 28,  
> > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > "initializing\_shards" : 2,  
> > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > }  
> > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > {  
> > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > "active\_primary\_shards" : 30,  
> > > > > > > > > > "active\_shards" : 57,  
> > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > }
> > > > > > > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > > {  
> > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > "active\_primary\_shards" : 45,  
> > > > > > > > > > "active\_shards" : 87,  
> > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > }  
> > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > > > > > > {  
> > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > "status" : "red",  
> > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > "active\_primary\_shards" : 59,  
> > > > > > > > > > "active\_shards" : 112,  
> > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > "initializing\_shards" : 7,  
> > > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > > }  
> > > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > > The detail logs of 20 nodes are attached.  
> > > > > > > > > > Thank you.
> > > > > > > > > > 
> > > > > > > > > > -Regards-  
> > > > > > > > > > -Dong Aihua-
> > > > > > > > > > 
> > > > > > > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > > > > > > 
> > > > > > > > > > > Hey,
> > > > > > > > > > > 
> > > > > > > > > > > thanks for testing this. I can see some exception during  
> > > > > > > > > > > recovery when starting a replica which causes one of the machines to wait  
> > > > > > > > > > > for an answer but it doesn't come back and it doesn't seem to get notified  
> > > > > > > > > > > that the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > > > > > > 30s"  
> > > > > > > > > > > So we can see if this happens just because of the closed  
> > > > > > > > > > > connections here?
> > > > > > > > > > > 
> > > > > > > > > > > simon
> > > > > > > > > > > 
> > > > > > > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > > > > > > 
> > > > > > > > > > > > Hi, Clint:  
> > > > > > > > > > > > I already opened an issue  
> > > > > > > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it  
> > > > > > > > > > > > has the same problem.  
> > > > > > > > > > > > My test configure is as following:  
> > > > > > > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > > > > > > My test step is as following:
> > > > > > > > > > > > 
> > > > > > > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > {  
> > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > > > > "active\_shards" : 27,  
> > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > > }  
> > > > > > > > > > > > The cluster stayed in this state for long time.
> > > > > > > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > {  
> > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > > > > > > "active\_shards" : 53,  
> > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > > > 3. Then I created one more empty index:  
> > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > {  
> > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > > > > > > "active\_shards" : 78,  
> > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > > > "unassigned\_shards" : 6  
> > > > > > > > > > > > }  
> > > > > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > > > > You can refer the detail logs from 20 nodes in the  
> > > > > > > > > > > > attachments.  
> > > > > > > > > > > > Thank you.
> > > > > > > > > > > > 
> > > > > > > > > > > > -Regards-  
> > > > > > > > > > > > -Dong Aihua-
> > > > > > > > > > > > 
> > > > > > > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > > > > > > 
> > > > > > > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hi, guys:  
> > > > > > > > > > > > > > These days through the test, I found ES bugs in 0.20.4 and  
> > > > > > > > > > > > > > 0.20.5  
> > > > > > > > > > > > > > which cause shards allocation failure and stuck in  
> > > > > > > > > > > > > > initializing state.  
> > > > > > > > > > > > > > The following are my test steps:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh cluster  
> > > > > > > > > > > > > > up. 3  
> > > > > > > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes  
> > > > > > > > > > > > > > are data  
> > > > > > > > > > > > > > nodes
> > > > > > > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > > > > > > indices for  
> > > > > > > > > > > > > > example index-2013-02-25, index-2013-02-26,  
> > > > > > > > > > > > > > index-2013-02-27,  
> > > > > > > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > > > > > > But some shards stuck in initializing status for long  
> > > > > > > > > > > > > > time.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Please can you open an issue on github, with a full recreation  
> > > > > > > > > > > > > of what  
> > > > > > > > > > > > > you need to do to recreate this problem, plus all the logs  
> > > > > > > > > > > > > from all of  
> > > > > > > > > > > > > the nodes.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > ta
> > > > > > > > > > > > > 
> > > > > > > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [April 1, 2013, 3:21am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/19 "2013-04-01T03:21:49Z")

</div>

Hi, Simon:  
I tested the latest trunk code, it also has no problem. The logs are  
located at [git@github.com](mailto:git@github.com):dongaihua/shares.git  
By the way, we ever met TooManyOpenFiles problem, but after we change the  
ulimit -n , we don't meet that problems again.  
After you check the logs, can you give me a confirmation if the problem  
is really resolved?  
Thank you.

在 2013年3月28日星期四UTC+8下午4时36分00秒，Dong Aihua写道：

> Ok, I will test it and tell you the result and logs.
> 
> 在 2013年3月28日星期四UTC+8下午4时07分17秒，simonw写道：
> 
> > Hmm that is interesting... I suspect that the fix for the missed  
> > ChannelClosed events fixed it then though....I don't have you logs around  
> > anymore but did you see any TooManyOpenFiles exceptions by any chance? Can  
> > you confirm that the latest master has fixed this as well by running a  
> > master version / latest build?
> > 
> > simon
> > 
> > On Thursday, March 28, 2013 3:44:10 AM UTC+1, Dong Aihua wrote:
> > 
> > > Hi, Simon:  
> > > Today I tested the latest ES version 0.20.6, the problem seems fixed  
> > > in this version.  
> > > Can you confirm it? And I'm wondering how do you fix it?  
> > > Thanks a lot.
> > > 
> > > -Regards-  
> > > -dongaihua-
> > > 
> > > 在 2013年3月11日星期一UTC+8下午6时10分08秒，Dong Aihua写道：
> > > 
> > > > Hi, Simon:  
> > > > Today I tested 0.19.12, it is fine.  
> > > > Then I tested 0.20.0, the problem happened again. So I guess the  
> > > > problem is introduced since 0.20.0.  
> > > > The test steps are as before. And the following are some logs:  
> > > > {  
> > > > "cluster\_name" : "test-0.20.0",  
> > > > "status" : "red",  
> > > > "timed\_out" : false,  
> > > > "number\_of\_nodes" : 20,  
> > > > "number\_of\_data\_nodes" : 15,  
> > > > "active\_primary\_shards" : 224,  
> > > > "active\_shards" : 445,  
> > > > "relocating\_shards" : 0,  
> > > > "initializing\_shards" : 4,  
> > > > "unassigned\_shards" : 1  
> > > > }  
> > > > { cluster\_name: 'test-0.20.0',  
> > > > master\_node:  
> > > > { name: 'xseed021.kdev',  
> > > > transport\_address: 'inet[/10.96.250.211:9400]',  
> > > > attributes: { data: 'false', master: 'true' } },  
> > > > initShards:  
> > > > [ { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> > > > primary: false,  
> > > > shard: 5,  
> > > > index: 'testx-xxx-2012-03-11' },  
> > > > { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> > > > primary: false,  
> > > > shard: 11,  
> > > > index: 'test-2013-03-11' },  
> > > > { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> > > > primary: true,  
> > > > shard: 7,  
> > > > index: 'testx-xxx-2013-zzz' },  
> > > > { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> > > > primary: false,  
> > > > shard: 5,  
> > > > index: 'testx-xxx-2013-yyy' } ],  
> > > > unassigned\_shards\_total: 1,  
> > > > unassigned\_indices: ['testx-xxx-2013-zzz'] }  
> > > > The detail logs are located at [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares/)  
> > > > test-0.20.0\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/test-0.20.0\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/test-0.20.0_LOGS.tar.gz)  
> > > > By the way, the servers are setup by IT team, I'm not clear the  
> > > > physical connections. If you really need those informaiton, I can ask and  
> > > > give you response.  
> > > > Thank you.
> > > > 
> > > > -Regards-  
> > > > -dongaihua-
> > > > 
> > > > 在 2013年3月11日星期一UTC+8下午3时56分07秒，simonw写道：
> > > > 
> > > > > Thanks for the headsup, that is what I figured. I am just wondering  
> > > > > what triggered the intermediate disconnects that causes the servers to wait  
> > > > > on their peers they recover from. Can you tell a little about your setup,  
> > > > > are they VMs do you have firewalls in place. Are the servers in different  
> > > > > datascenters, what is the connection between them?
> > > > > 
> > > > > simon
> > > > > 
> > > > > On Monday, March 11, 2013 3:14:01 AM UTC+1, Dong Aihua wrote:
> > > > > 
> > > > > > By the way, for the servers disconnect at the end of logs, the reason  
> > > > > > is I did a shutdown command after the test finish
> > > > > > 
> > > > > > 在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：
> > > > > > 
> > > > > > > So, I looked closer at the latest logs and I see a lot of  
> > > > > > > disconnects going on giving me the impression you have some network issues.  
> > > > > > > Nevertheless we pushed some stuff to detect these situations earlier but  
> > > > > > > non of us was able to reproduce your issues. the only thing I can ask you  
> > > > > > > for is to try again with latest master to see if those commits helped in  
> > > > > > > any way?  
> > > > > > > What is your setup by the way, any idea why servers disconnect all  
> > > > > > > the time?
> > > > > > > 
> > > > > > > simon
> > > > > > > 
> > > > > > > On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> > > > > > > 
> > > > > > > > Hi, :  
> > > > > > > > Is there any clue for this problem's root cause?  
> > > > > > > > Thank you.
> > > > > > > > 
> > > > > > > > -Regards-  
> > > > > > > > -Dong Aihua-
> > > > > > > > 
> > > > > > > > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > > > > > > > 
> > > > > > > > > Hi, Simon  
> > > > > > > > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > > > > > > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > > > > > > > Thank you.
> > > > > > > > > 
> > > > > > > > > -Regards-  
> > > > > > > > > -Dong aihua-
> > > > > > > > > 
> > > > > > > > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > > > > > > > 
> > > > > > > > > > I uploaded several times for the logs. All failed, I got the 340  
> > > > > > > > > > error. I will upload the logs later.
> > > > > > > > > > 
> > > > > > > > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > > > > > > > 
> > > > > > > > > > > Hi, Simon:  
> > > > > > > > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > > > > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > > > > > > > The same problem happened the again.  
> > > > > > > > > > > The configuration is same as before. The following are my test  
> > > > > > > > > > > steps:
> > > > > > > > > > > 
> > > > > > > > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > > {  
> > > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > > > "active\_shards" : 28,  
> > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > "initializing\_shards" : 2,  
> > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > }  
> > > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > > {  
> > > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > "active\_primary\_shards" : 30,  
> > > > > > > > > > > "active\_shards" : 57,  
> > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > }
> > > > > > > > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > > > {  
> > > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > "active\_primary\_shards" : 45,  
> > > > > > > > > > > "active\_shards" : 87,  
> > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > }  
> > > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > > > > > > > {  
> > > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > "active\_primary\_shards" : 59,  
> > > > > > > > > > > "active\_shards" : 112,  
> > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > "initializing\_shards" : 7,  
> > > > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > > > }  
> > > > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > > > The detail logs of 20 nodes are attached.  
> > > > > > > > > > > Thank you.
> > > > > > > > > > > 
> > > > > > > > > > > -Regards-  
> > > > > > > > > > > -Dong Aihua-
> > > > > > > > > > > 
> > > > > > > > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > > > > > > > 
> > > > > > > > > > > > Hey,
> > > > > > > > > > > > 
> > > > > > > > > > > > thanks for testing this. I can see some exception during  
> > > > > > > > > > > > recovery when starting a replica which causes one of the machines to wait  
> > > > > > > > > > > > for an answer but it doesn't come back and it doesn't seem to get notified  
> > > > > > > > > > > > that the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > > > > > > > 30s"  
> > > > > > > > > > > > So we can see if this happens just because of the closed  
> > > > > > > > > > > > connections here?
> > > > > > > > > > > > 
> > > > > > > > > > > > simon
> > > > > > > > > > > > 
> > > > > > > > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hi, Clint:  
> > > > > > > > > > > > > I already opened an issue  
> > > > > > > > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it  
> > > > > > > > > > > > > has the same problem.  
> > > > > > > > > > > > > My test configure is as following:  
> > > > > > > > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is the  
> > > > > > > > > > > > > leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > > > > > > > My test step is as following:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > > {  
> > > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > > > > > "active\_shards" : 27,  
> > > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > > > }  
> > > > > > > > > > > > > The cluster stayed in this state for long time.
> > > > > > > > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > > {  
> > > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > > > > > > > "active\_shards" : 53,  
> > > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > > > > 3. Then I created one more empty index:  
> > > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > > {  
> > > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > > > > > > > "active\_shards" : 78,  
> > > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > > > > "unassigned\_shards" : 6  
> > > > > > > > > > > > > }  
> > > > > > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > > > > > You can refer the detail logs from 20 nodes in the  
> > > > > > > > > > > > > attachments.  
> > > > > > > > > > > > > Thank you.
> > > > > > > > > > > > > 
> > > > > > > > > > > > > -Regards-  
> > > > > > > > > > > > > -Dong Aihua-
> > > > > > > > > > > > > 
> > > > > > > > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Hi, guys:  
> > > > > > > > > > > > > > > These days through the test, I found ES bugs in 0.20.4  
> > > > > > > > > > > > > > > and 0.20.5  
> > > > > > > > > > > > > > > which cause shards allocation failure and stuck in  
> > > > > > > > > > > > > > > initializing state.  
> > > > > > > > > > > > > > > The following are my test steps:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh  
> > > > > > > > > > > > > > > cluster up. 3  
> > > > > > > > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15 nodes  
> > > > > > > > > > > > > > > are data  
> > > > > > > > > > > > > > > nodes
> > > > > > > > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > > > > > > > indices for  
> > > > > > > > > > > > > > > example index-2013-02-25, index-2013-02-26,  
> > > > > > > > > > > > > > > index-2013-02-27,  
> > > > > > > > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > > > > > > > But some shards stuck in initializing status for long  
> > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Please can you open an issue on github, with a full  
> > > > > > > > > > > > > > recreation of what  
> > > > > > > > > > > > > > you need to do to recreate this problem, plus all the logs  
> > > > > > > > > > > > > > from all of  
> > > > > > > > > > > > > > the nodes.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > ta
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Dong\_Aihua](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dong_aihua/32/2413_2.png) [@Dong\_Aihua](https://discuss.elastic.co/u/Dong_Aihua)\
**Post date:** [April 11, 2013, 6:00am UTC](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915/20 "2013-04-11T06:00:56Z")

</div>

Hi, Simon:  
Today, I upgraded the es cluster from 0.19.11 to 0.20.6 successfully. The  
upgrade is smooth.  
And this problem doesn't see again.  
Thank you.

在 2013年4月1日星期一UTC+8上午11时21分49秒，Dong Aihua写道：

> Hi, Simon:  
> I tested the latest trunk code, it also has no problem. The logs are  
> located at [git@github.com](mailto:git@github.com):dongaihua/shares.git  
> By the way, we ever met TooManyOpenFiles problem, but after we change  
> the ulimit -n , we don't meet that problems again.  
> After you check the logs, can you give me a confirmation if the problem  
> is really resolved?  
> Thank you.
> 
> 在 2013年3月28日星期四UTC+8下午4时36分00秒，Dong Aihua写道：
> 
> > Ok, I will test it and tell you the result and logs.
> > 
> > 在 2013年3月28日星期四UTC+8下午4时07分17秒，simonw写道：
> > 
> > > Hmm that is interesting... I suspect that the fix for the missed  
> > > ChannelClosed events fixed it then though....I don't have you logs around  
> > > anymore but did you see any TooManyOpenFiles exceptions by any chance? Can  
> > > you confirm that the latest master has fixed this as well by running a  
> > > master version / latest build?
> > > 
> > > simon
> > > 
> > > On Thursday, March 28, 2013 3:44:10 AM UTC+1, Dong Aihua wrote:
> > > 
> > > > Hi, Simon:  
> > > > Today I tested the latest ES version 0.20.6, the problem seems fixed  
> > > > in this version.  
> > > > Can you confirm it? And I'm wondering how do you fix it?  
> > > > Thanks a lot.
> > > > 
> > > > -Regards-  
> > > > -dongaihua-
> > > > 
> > > > 在 2013年3月11日星期一UTC+8下午6时10分08秒，Dong Aihua写道：
> > > > 
> > > > > Hi, Simon:  
> > > > > Today I tested 0.19.12, it is fine.  
> > > > > Then I tested 0.20.0, the problem happened again. So I guess the  
> > > > > problem is introduced since 0.20.0.  
> > > > > The test steps are as before. And the following are some logs:  
> > > > > {  
> > > > > "cluster\_name" : "test-0.20.0",  
> > > > > "status" : "red",  
> > > > > "timed\_out" : false,  
> > > > > "number\_of\_nodes" : 20,  
> > > > > "number\_of\_data\_nodes" : 15,  
> > > > > "active\_primary\_shards" : 224,  
> > > > > "active\_shards" : 445,  
> > > > > "relocating\_shards" : 0,  
> > > > > "initializing\_shards" : 4,  
> > > > > "unassigned\_shards" : 1  
> > > > > }  
> > > > > { cluster\_name: 'test-0.20.0',  
> > > > > master\_node:  
> > > > > { name: 'xseed021.kdev',  
> > > > > transport\_address: 'inet[/10.96.250.211:9400]',  
> > > > > attributes: { data: 'false', master: 'true' } },  
> > > > > initShards:  
> > > > > [ { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> > > > > primary: false,  
> > > > > shard: 5,  
> > > > > index: 'testx-xxx-2012-03-11' },  
> > > > > { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> > > > > primary: false,  
> > > > > shard: 11,  
> > > > > index: 'test-2013-03-11' },  
> > > > > { node: '{"name":"xseed034.kdev","ip":"10.96.250.224:9400"}',  
> > > > > primary: true,  
> > > > > shard: 7,  
> > > > > index: 'testx-xxx-2013-zzz' },  
> > > > > { node: '{"name":"xseed038.kdev","ip":"10.96.250.228:9400"}',  
> > > > > primary: false,  
> > > > > shard: 5,  
> > > > > index: 'testx-xxx-2013-yyy' } ],  
> > > > > unassigned\_shards\_total: 1,  
> > > > > unassigned\_indices: ['testx-xxx-2013-zzz'] }  
> > > > > The detail logs are located at [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares/)  
> > > > > test-0.20.0\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/test-0.20.0\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/test-0.20.0_LOGS.tar.gz)  
> > > > > By the way, the servers are setup by IT team, I'm not clear the  
> > > > > physical connections. If you really need those informaiton, I can ask and  
> > > > > give you response.  
> > > > > Thank you.
> > > > > 
> > > > > -Regards-  
> > > > > -dongaihua-
> > > > > 
> > > > > 在 2013年3月11日星期一UTC+8下午3时56分07秒，simonw写道：
> > > > > 
> > > > > > Thanks for the headsup, that is what I figured. I am just wondering  
> > > > > > what triggered the intermediate disconnects that causes the servers to wait  
> > > > > > on their peers they recover from. Can you tell a little about your setup,  
> > > > > > are they VMs do you have firewalls in place. Are the servers in different  
> > > > > > datascenters, what is the connection between them?
> > > > > > 
> > > > > > simon
> > > > > > 
> > > > > > On Monday, March 11, 2013 3:14:01 AM UTC+1, Dong Aihua wrote:
> > > > > > 
> > > > > > > By the way, for the servers disconnect at the end of logs, the  
> > > > > > > reason is I did a shutdown command after the test finish
> > > > > > > 
> > > > > > > 在 2013年3月7日星期四UTC+8下午7时04分22秒，simonw写道：
> > > > > > > 
> > > > > > > > So, I looked closer at the latest logs and I see a lot of  
> > > > > > > > disconnects going on giving me the impression you have some network issues.  
> > > > > > > > Nevertheless we pushed some stuff to detect these situations earlier but  
> > > > > > > > non of us was able to reproduce your issues. the only thing I can ask you  
> > > > > > > > for is to try again with latest master to see if those commits helped in  
> > > > > > > > any way?  
> > > > > > > > What is your setup by the way, any idea why servers disconnect all  
> > > > > > > > the time?
> > > > > > > > 
> > > > > > > > simon
> > > > > > > > 
> > > > > > > > On Thursday, March 7, 2013 3:11:57 AM UTC+1, Dong Aihua wrote:
> > > > > > > > 
> > > > > > > > > Hi, :  
> > > > > > > > > Is there any clue for this problem's root cause?  
> > > > > > > > > Thank you.
> > > > > > > > > 
> > > > > > > > > -Regards-  
> > > > > > > > > -Dong Aihua-
> > > > > > > > > 
> > > > > > > > > 在 2013年3月5日星期二UTC+8下午2时42分55秒，Dong Aihua写道：
> > > > > > > > > 
> > > > > > > > > > Hi, Simon  
> > > > > > > > > > Please get the logs es-test2-0-90-0-beta1\_LOGS.tar.gz[https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1\_LOGS.tar.gz](https://github.com/dongaihua/shares/blob/master/es-test2-0-90-0-beta1_LOGS.tar.gz)  
> > > > > > > > > > from [GitHub - dongaihua/shares: share something](https://github.com/dongaihua/shares)  
> > > > > > > > > > Thank you.
> > > > > > > > > > 
> > > > > > > > > > -Regards-  
> > > > > > > > > > -Dong aihua-
> > > > > > > > > > 
> > > > > > > > > > 在 2013年3月5日星期二UTC+9下午2时43分15秒，Dong Aihua写道：
> > > > > > > > > > 
> > > > > > > > > > > I uploaded several times for the logs. All failed, I got the 340  
> > > > > > > > > > > error. I will upload the logs later.
> > > > > > > > > > > 
> > > > > > > > > > > 在 2013年3月5日星期二UTC+9下午2时21分42秒，Dong Aihua写道：
> > > > > > > > > > > 
> > > > > > > > > > > > Hi, Simon:  
> > > > > > > > > > > > I tested the ES version 0.90.0.Beta1 again with the setting  
> > > > > > > > > > > > indices.recovery.internal\_action\_timeout: 30s.  
> > > > > > > > > > > > The same problem happened the again.  
> > > > > > > > > > > > The configuration is same as before. The following are my  
> > > > > > > > > > > > test steps:
> > > > > > > > > > > > 
> > > > > > > > > > > > 1. curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > > > {  
> > > > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > > > > "active\_shards" : 28,  
> > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > "initializing\_shards" : 2,  
> > > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > > }  
> > > > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > > > 2. curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > > > {  
> > > > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > "active\_primary\_shards" : 30,  
> > > > > > > > > > > > "active\_shards" : 57,  
> > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > > }
> > > > > > > > > > > > 3. curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > > > > {  
> > > > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > "active\_primary\_shards" : 45,  
> > > > > > > > > > > > "active\_shards" : 87,  
> > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > > }  
> > > > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > > > 4. curl -XPUT 10.96.250.214:10200/xxxxxyyyyy111  
> > > > > > > > > > > > {  
> > > > > > > > > > > > "cluster\_name" : "es-test2-0-90-0-beta1",  
> > > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > "active\_primary\_shards" : 59,  
> > > > > > > > > > > > "active\_shards" : 112,  
> > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > "initializing\_shards" : 7,  
> > > > > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > > > > }  
> > > > > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > > > > The detail logs of 20 nodes are attached.  
> > > > > > > > > > > > Thank you.
> > > > > > > > > > > > 
> > > > > > > > > > > > -Regards-  
> > > > > > > > > > > > -Dong Aihua-
> > > > > > > > > > > > 
> > > > > > > > > > > > 在 2013年3月4日星期一UTC+8下午4时47分52秒，simonw写道：
> > > > > > > > > > > > 
> > > > > > > > > > > > > Hey,
> > > > > > > > > > > > > 
> > > > > > > > > > > > > thanks for testing this. I can see some exception during  
> > > > > > > > > > > > > recovery when starting a replica which causes one of the machines to wait  
> > > > > > > > > > > > > for an answer but it doesn't come back and it doesn't seem to get notified  
> > > > > > > > > > > > > that the connection is closed. Can you try to set: "indices.recovery.internal\_action\_timeout:  
> > > > > > > > > > > > > 30s"  
> > > > > > > > > > > > > So we can see if this happens just because of the closed  
> > > > > > > > > > > > > connections here?
> > > > > > > > > > > > > 
> > > > > > > > > > > > > simon
> > > > > > > > > > > > > 
> > > > > > > > > > > > > On Monday, March 4, 2013 7:22:50 AM UTC+1, Dong Aihua wrote:
> > > > > > > > > > > > > 
> > > > > > > > > > > > > > Hi, Clint:  
> > > > > > > > > > > > > > I already opened an issue  
> > > > > > > > > > > > > > [ES have bugs in 0.20.4 and 0.20.5 which cause creating some indices failure. · Issue #2714 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2714)  
> > > > > > > > > > > > > > Today I tested the latest the ES version 0.90.0.Beta1, it  
> > > > > > > > > > > > > > has the same problem.  
> > > > > > > > > > > > > > My test configure is as following:  
> > > > > > > > > > > > > > 20 nodes. 10.96.250.211,212,213 are master nodes. 211 is  
> > > > > > > > > > > > > > the leader; 214 and 215 are load balancer; other 15 nodes are data nodes.  
> > > > > > > > > > > > > > My test step is as following:
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 1. after the cluster is up, I created an empty index: test1  
> > > > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/test1  
> > > > > > > > > > > > > > {"ok":true,"acknowledged":true}  
> > > > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > > > {  
> > > > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > > > "status" : "yellow",  
> > > > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > > > "active\_primary\_shards" : 15,  
> > > > > > > > > > > > > > "active\_shards" : 27,  
> > > > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > > > "initializing\_shards" : 3,  
> > > > > > > > > > > > > > "unassigned\_shards" : 0  
> > > > > > > > > > > > > > }  
> > > > > > > > > > > > > > The cluster stayed in this state for long time.
> > > > > > > > > > > > > > 2. Then I created another empty index: abcd1234  
> > > > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/abcd1234  
> > > > > > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > > > {  
> > > > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > > > "active\_primary\_shards" : 29,  
> > > > > > > > > > > > > > "active\_shards" : 53,  
> > > > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > > > > > "unassigned\_shards" : 1  
> > > > > > > > > > > > > > The cluster stayed in this state for long time
> > > > > > > > > > > > > > 3. Then I created one more empty index:  
> > > > > > > > > > > > > > curl -XPUT 10.96.250.214:10200/1234abcd  
> > > > > > > > > > > > > > {"ok":true,"acknowledged":false}  
> > > > > > > > > > > > > > curl -XGET 10.96.250.211:10200/\_cluster/health?pretty=true  
> > > > > > > > > > > > > > {  
> > > > > > > > > > > > > > "cluster\_name" : "es-test-0-90-0-beta1",  
> > > > > > > > > > > > > > "status" : "red",  
> > > > > > > > > > > > > > "timed\_out" : false,  
> > > > > > > > > > > > > > "number\_of\_nodes" : 20,  
> > > > > > > > > > > > > > "number\_of\_data\_nodes" : 15,  
> > > > > > > > > > > > > > "active\_primary\_shards" : 43,  
> > > > > > > > > > > > > > "active\_shards" : 78,  
> > > > > > > > > > > > > > "relocating\_shards" : 0,  
> > > > > > > > > > > > > > "initializing\_shards" : 6,  
> > > > > > > > > > > > > > "unassigned\_shards" : 6  
> > > > > > > > > > > > > > }  
> > > > > > > > > > > > > > The cluster stayed in this state for long time.  
> > > > > > > > > > > > > > You can refer the detail logs from 20 nodes in the  
> > > > > > > > > > > > > > attachments.  
> > > > > > > > > > > > > > Thank you.
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > -Regards-  
> > > > > > > > > > > > > > -Dong Aihua-
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > 在 2013年2月27日星期三UTC+8下午6时51分49秒，Clinton Gormley写道：
> > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > On Tue, 2013-02-26 at 16:09 -0800, jackiedong wrote:
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > Hi, guys:  
> > > > > > > > > > > > > > > > These days through the test, I found ES bugs in 0.20.4  
> > > > > > > > > > > > > > > > and 0.20.5  
> > > > > > > > > > > > > > > > which cause shards allocation failure and stuck in  
> > > > > > > > > > > > > > > > initializing state.  
> > > > > > > > > > > > > > > > The following are my test steps:
> > > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > > 1. I setup 20 nodes with 0.20.4, and bring a fresh  
> > > > > > > > > > > > > > > > cluster up. 3  
> > > > > > > > > > > > > > > > nodes are master nodes, 2 nodes are load balancer, 15  
> > > > > > > > > > > > > > > > nodes are data  
> > > > > > > > > > > > > > > > nodes
> > > > > > > > > > > > > > > > 2. After the cluster is up, I tried to create some empty  
> > > > > > > > > > > > > > > > indices for  
> > > > > > > > > > > > > > > > example index-2013-02-25, index-2013-02-26,  
> > > > > > > > > > > > > > > > index-2013-02-27,  
> > > > > > > > > > > > > > > > index-2013-03-01, etc  
> > > > > > > > > > > > > > > > But some shards stuck in initializing status for long  
> > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Please can you open an issue on github, with a full  
> > > > > > > > > > > > > > > recreation of what  
> > > > > > > > > > > > > > > you need to do to recreate this problem, plus all the logs  
> > > > > > > > > > > > > > > from all of  
> > > > > > > > > > > > > > > the nodes.
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > ta
> > > > > > > > > > > > > > > 
> > > > > > > > > > > > > > > Clint

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

[Next page](https://discuss.elastic.co/t/es-bugs-in-0-20-4-and-0-20-5-cause-shards-allocation-failure-and-stuck-in-initializing-state/10915.md?page=2)