# Cluster health issues with 0.9.0-SNAPSHOT **URL:** \ **Category:** Elasticsearch\ **Created:** [July 16, 2010, 12:14pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114 "2010-07-16T12:14:42Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![Stephen\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephen_2/32/3274_2.png) [@Stephen\_2](https://discuss.elastic.co/u/Stephen_2)\ **Post date:** [July 16, 2010, 12:14pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/1 "2010-07-16T12:14:42Z")
Hey guys, I've been doing some fairly ornate integration testing with my app and ES, and have hit an interesting snag. What I'm doing is creating three nodes and shoving a bunch of data into it, but with a twist -- some of my indexes are empty (as in, the index exists but has no documents in it yet). This state is necessary for my application since people can create new projects without putting any data into them and still expect their global searches to work. If I don't create empty indexes (as in, client.admin().indices().prepareCreate(absoluteName).execute().actionGet();), then all my searches fail due to shard failures. So, the side effect of having empty indexes is that if one of my nodes goes down, the cluster status goes yellow and stays that way. It seems that the indexes that were NOT empty get replicated properly, but the ones that ARE empty get ignored. This also causes very strange recovery failures when I shut down a second node, which I'm hoping is related to the emptiness problem. Here's what my work directory looks like with all three nodes happy: starkey-imac:nodes stephen$ du -sh \* 1.1M 0 1.2M 1 1.1M 2 starkey-imac:~ stephen$ curl -XGET '[http://192.168.0.5:9200/\_cluster/](http://192.168.0.5:9200/_cluster/) health' {"status":"green","timed\_out":false,"active\_primary\_shards": 220,"active\_shards":440,"relocating\_shards":0}s And when I kill one of them: starkey-imac:nodes stephen$ du -sh \* 1.5M 0 1.2M 1 0B 2 starkey-imac:~ stephen$ curl -XGET '[http://192.168.0.5:9200/\_cluster/](http://192.168.0.5:9200/_cluster/) health' {"status":"yellow","timed\_out":false,"active\_primary\_shards": 220,"active\_shards":359,"relocating\_shards":0} The problem is, I'm kind of paranoid -- I refuse to allow my application to write into the cluster unless it's green. However, this seriously puts a crimp in my style, since the cluster never goes green after this happens. Any thoughts? Thanks in advance, Stephen. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 16, 2010, 6:04pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/2 "2010-07-16T18:04:37Z")
I've ran a simple test where I created a single empty index on a three node cluster, and the cluster health was green. Then, I shutdown one node, cluster health is still green. Then I shutdown the second node (which results in one node left) and health is yellow (as expected). Is there a chance for a recreation? -shay.banon On Fri, Jul 16, 2010 at 3:14 PM, Stephen [scstarkey@gmail.com](mailto:scstarkey@gmail.com) wrote: > Hey guys, > > I've been doing some fairly ornate integration testing with my app and > ES, and have hit an interesting snag. What I'm doing is creating > three nodes and shoving a bunch of data into it, but with a twist -- > some of my indexes are empty (as in, the index exists but has no > documents in it yet). > > This state is necessary for my application since people can create new > projects without putting any data into them and still expect their > global searches to work. If I don't create empty indexes (as in, > > client.admin().indices().prepareCreate(absoluteName).execute().actionGet();), > then all my searches fail due to shard failures. > > So, the side effect of having empty indexes is that if one of my nodes > goes down, the cluster status goes yellow and stays that way. It > seems that the indexes that were NOT empty get replicated properly, > but the ones that ARE empty get ignored. This also causes very > strange recovery failures when I shut down a second node, which I'm > hoping is related to the emptiness problem. > > Here's what my work directory looks like with all three nodes happy: > > starkey-imac:nodes stephen$ du -sh \* > 1.1M 0 > 1.2M 1 > 1.1M 2 > > starkey-imac:~ stephen$ curl -XGET '[http://192.168.0.5:9200/\_cluster/](http://192.168.0.5:9200/_cluster/) > health' > {"status":"green","timed\_out":false,"active\_primary\_shards": > 220,"active\_shards":440,"relocating\_shards":0}s > > And when I kill one of them: > > starkey-imac:nodes stephen$ du -sh \* > 1.5M 0 > 1.2M 1 > 0B 2 > > starkey-imac:~ stephen$ curl -XGET '[http://192.168.0.5:9200/\_cluster/](http://192.168.0.5:9200/_cluster/) > health' > {"status":"yellow","timed\_out":false,"active\_primary\_shards": > 220,"active\_shards":359,"relocating\_shards":0} > > The problem is, I'm kind of paranoid -- I refuse to allow my > application to write into the cluster unless it's green. However, > this seriously puts a crimp in my style, since the cluster never goes > green after this happens. > > Any thoughts? > > Thanks in advance, > > Stephen. ---
**Author:** ![Stephen\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephen_2/32/3274_2.png) [@Stephen\_2](https://discuss.elastic.co/u/Stephen_2)\ **Post date:** [July 17, 2010, 12:37pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/3 "2010-07-17T12:37:17Z")
Unfortunately, the only thing I can do to reproduce it with simple data requires a lot of trial and error. That and more often than not I run out of memory if I play with it too much. I would need a much more powerful machine to get it to work right. All I can say is it seems to happen reliably when I have 3 nodes, and alternate between turning one of them off and on again in quick succession. Perhaps the problem is I don't leave time to allow them to replicate back. But, I could swear I waited for all the indexes to turn green again before killing the next node. I'll play with it more. Stephen. On Jul 16, 1:04 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > I've ran a simple test where I created a single empty index on a three node > cluster, and the cluster health was green. Then, I shutdown one node, > cluster health is still green. Then I shutdown the second node (which > results in one node left) and health is yellow (as expected). > > Is there a chance for a recreation? > > -shay.banon > > On Fri, Jul 16, 2010 at 3:14 PM, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > Hey guys, > > > I've been doing some fairly ornate integration testing with my app and > > ES, and have hit an interesting snag. What I'm doing is creating > > three nodes and shoving a bunch of data into it, but with a twist -- > > some of my indexes are empty (as in, the index exists but has no > > documents in it yet). > > > This state is necessary for my application since people can create new > > projects without putting any data into them and still expect their > > global searches to work. If I don't create empty indexes (as in, > > > client.admin().indices().prepareCreate(absoluteName).execute().actionGet(); ), > > then all my searches fail due to shard failures. > > > So, the side effect of having empty indexes is that if one of my nodes > > goes down, the cluster status goes yellow and stays that way. It > > seems that the indexes that were NOT empty get replicated properly, > > but the ones that ARE empty get ignored. This also causes very > > strange recovery failures when I shut down a second node, which I'm > > hoping is related to the emptiness problem. > > > Here's what my work directory looks like with all three nodes happy: > > > starkey-imac:nodes stephen$ du -sh \* > > 1.1M 0 > > 1.2M 1 > > 1.1M 2 > > > starkey-imac:~ stephen$ curl -XGET '[http://192.168.0.5:9200/\_cluster/](http://192.168.0.5:9200/_cluster/) > > health' > > {"status":"green","timed\_out":false,"active\_primary\_shards": > > 220,"active\_shards":440,"relocating\_shards":0}s > > > And when I kill one of them: > > > starkey-imac:nodes stephen$ du -sh \* > > 1.5M 0 > > 1.2M 1 > > 0B 2 > > > starkey-imac:~ stephen$ curl -XGET '[http://192.168.0.5:9200/\_cluster/](http://192.168.0.5:9200/_cluster/) > > health' > > {"status":"yellow","timed\_out":false,"active\_primary\_shards": > > 220,"active\_shards":359,"relocating\_shards":0} > > > The problem is, I'm kind of paranoid -- I refuse to allow my > > application to write into the cluster unless it's green. However, > > this seriously puts a crimp in my style, since the cluster never goes > > green after this happens. > > > Any thoughts? > > > Thanks in advance, > > > Stephen. ---
**Author:** ![Stephen\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephen_2/32/3274_2.png) [@Stephen\_2](https://discuss.elastic.co/u/Stephen_2)\ **Post date:** [July 17, 2010, 4:14pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/4 "2010-07-17T16:14:05Z")
Okay, So I've spent all morning writing a fairly involved test class which starts up its own nodes. I then ran a series of test cases that seem to reproduce the health issue fairly regularly. The methodology I used to execute the tests is as follows: 1. Vary the number of nodes from 2 to 4 2. Vary the number of indexes between 5 and 20 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% (alldata) 4. When the number of nodes is greater than 2, and all indexes are GREEN, and then kill the LAST node 5. Before declaring the test to be a failure, wait for the log to stop moving (after waiting 15-30 seconds) 6. Used default values for everything -- no special configuration of indexes, replication, shard counts, etc. 7. In the case where the indexes never turn green, re-run the test a couple times 8. Allocated -Xmx1024m to the entire test case Assumptions: 1. A cluster should always strive toward GREEN when there is more than one node 2. If a node is lost, the cluster should always rebalance until it becomes GREEN again (assuming more than one node) Results: In many cases, ES fails to maintain appropriate balance. It has most success when there are 5 indexes, but still not perfect. As the number of nodes increase, the odds for failure increase on node failure. Also, as the number of indexes increase, the odds for failure increase. Test Class: [http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) Detailed Results: [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQVFMWXdBN01lNXc&hl=en&pli=1#gid=0](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQVFMWXdBN01lNXc&hl=en&pli=1#gid=0) The only dependencies I can see are the ones necessary to build elasticsearch. Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes to create] [nodata, somedata, or alldata] ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 17, 2010, 8:29pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/5 "2010-07-17T20:29:18Z")
Hi, Thanks for the test. I can't access your spreadsheet, but I managed to recreate it locally (happens very sporadically). Will look into this and update you once I fix it. -shay.banon On Sat, Jul 17, 2010 at 7:14 PM, Stephen [scstarkey@gmail.com](mailto:scstarkey@gmail.com) wrote: > Okay, > > So I've spent all morning writing a fairly involved test class which > starts up its own nodes. I then ran a series of test cases that seem > to reproduce the health issue fairly regularly. The methodology I > used to execute the tests is as follows: > > 1. Vary the number of nodes from 2 to 4 > 2. Vary the number of indexes between 5 and 20 > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% > (alldata) > 4. When the number of nodes is greater than 2, and all indexes are > GREEN, and then kill the LAST node > 5. Before declaring the test to be a failure, wait for the log to stop > moving (after waiting 15-30 seconds) > 6. Used default values for everything -- no special configuration of > indexes, replication, shard counts, etc. > 7. In the case where the indexes never turn green, re-run the test a > couple times > 8. Allocated -Xmx1024m to the entire test case > > Assumptions: > > 1. A cluster should always strive toward GREEN when there is more than > one node > 2. If a node is lost, the cluster should always rebalance until it > becomes GREEN again (assuming more than one node) > > Results: > > In many cases, ES fails to maintain appropriate balance. It has most > success when there are 5 indexes, but still not perfect. As the > number of nodes increase, the odds for failure increase on node > failure. Also, as the number of indexes increase, the odds for > failure increase. > > Test Class: [http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) > Detailed Results: > [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQVFMWXdBN01lNXc&hl=en&pli=1#gid=0](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQVFMWXdBN01lNXc&hl=en&pli=1#gid=0) > > The only dependencies I can see are the ones necessary to build > elasticsearch. > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes to > create] [nodata, somedata, or alldata] ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 18, 2010, 7:55pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/6 "2010-07-18T19:55:34Z")
Hi, I just pushed a fix for this, I can't get it to happen with your test. If you can verify it, it would be great. I will write some built in similar tests for this that will run as part of the test suite of elasticsearch. -shay.banon On Sat, Jul 17, 2010 at 11:29 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > Hi, > > Thanks for the test. I can't access your spreadsheet, but I managed to > recreate it locally (happens very sporadically). Will look into this and > update you once I fix it. > > -shay.banon > > On Sat, Jul 17, 2010 at 7:14 PM, Stephen [scstarkey@gmail.com](mailto:scstarkey@gmail.com) wrote: > > > Okay, > > > > So I've spent all morning writing a fairly involved test class which > > starts up its own nodes. I then ran a series of test cases that seem > > to reproduce the health issue fairly regularly. The methodology I > > used to execute the tests is as follows: > > > > 1. Vary the number of nodes from 2 to 4 > > 2. Vary the number of indexes between 5 and 20 > > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% > > (alldata) > > 4. When the number of nodes is greater than 2, and all indexes are > > GREEN, and then kill the LAST node > > 5. Before declaring the test to be a failure, wait for the log to stop > > moving (after waiting 15-30 seconds) > > 6. Used default values for everything -- no special configuration of > > indexes, replication, shard counts, etc. > > 7. In the case where the indexes never turn green, re-run the test a > > couple times > > 8. Allocated -Xmx1024m to the entire test case > > > > Assumptions: > > > > 1. A cluster should always strive toward GREEN when there is more than > > one node > > 2. If a node is lost, the cluster should always rebalance until it > > becomes GREEN again (assuming more than one node) > > > > Results: > > > > In many cases, ES fails to maintain appropriate balance. It has most > > success when there are 5 indexes, but still not perfect. As the > > number of nodes increase, the odds for failure increase on node > > failure. Also, as the number of indexes increase, the odds for > > failure increase. > > > > Test Class: [http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) > > Detailed Results: > > [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQVFMWXdBN01lNXc&hl=en&pli=1#gid=0](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQVFMWXdBN01lNXc&hl=en&pli=1#gid=0) > > > > The only dependencies I can see are the ones necessary to build > > elasticsearch. > > > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes to > > create] [nodata, somedata, or alldata] ---
**Author:** ![Stephen\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephen_2/32/3274_2.png) [@Stephen\_2](https://discuss.elastic.co/u/Stephen_2)\ **Post date:** [July 20, 2010, 11:19am UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/7 "2010-07-20T11:19:55Z")
... is there a chance anybody's looked at this? On Jul 17, 11:14 am, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > Okay, > > So I've spent all morning writing a fairly involved test class which > starts up its own nodes. I then ran a series of test cases that seem > to reproduce the health issue fairly regularly. The methodology I > used to execute the tests is as follows: > > 1. Vary the number of nodes from 2 to 4 > 2. Vary the number of indexes between 5 and 20 > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% > (alldata) > 4. When the number of nodes is greater than 2, and all indexes are > GREEN, and then kill the LAST node > 5. Before declaring the test to be a failure, wait for the log to stop > moving (after waiting 15-30 seconds) > 6. Used default values for everything -- no special configuration of > indexes, replication, shard counts, etc. > 7. In the case where the indexes never turn green, re-run the test a > couple times > 8. Allocated -Xmx1024m to the entire test case > > Assumptions: > > 1. A cluster should always strive toward GREEN when there is more than > one node > 2. If a node is lost, the cluster should always rebalance until it > becomes GREEN again (assuming more than one node) > > Results: > > In many cases, ES fails to maintain appropriate balance. It has most > success when there are 5 indexes, but still not perfect. As the > number of nodes increase, the odds for failure increase on node > failure. Also, as the number of indexes increase, the odds for > failure increase. > > Test Class:[http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) > Detailed Results:[http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ)... > > The only dependencies I can see are the ones necessary to build > elasticsearch. > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes to > create] [nodata, somedata, or alldata] ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 20, 2010, 1:48pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/8 "2010-07-20T13:48:23Z")
Maybe you missed my previous email from Jul 18, here it is again 🙂 : " Hi, I just pushed a fix for this, I can't get it to happen with your test anymore. If you can verify it, it would be great. I will write some built in similar tests for this that will run as part of the test suite of elasticsearch. " -shay.banon On Tue, Jul 20, 2010 at 2:19 PM, Stephen [scstarkey@gmail.com](mailto:scstarkey@gmail.com) wrote: > ... is there a chance anybody's looked at this? > > On Jul 17, 11:14 am, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > Okay, > > > > So I've spent all morning writing a fairly involved test class which > > starts up its own nodes. I then ran a series of test cases that seem > > to reproduce the health issue fairly regularly. The methodology I > > used to execute the tests is as follows: > > > > 1. Vary the number of nodes from 2 to 4 > > 2. Vary the number of indexes between 5 and 20 > > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% > > (alldata) > > 4. When the number of nodes is greater than 2, and all indexes are > > GREEN, and then kill the LAST node > > 5. Before declaring the test to be a failure, wait for the log to stop > > moving (after waiting 15-30 seconds) > > 6. Used default values for everything -- no special configuration of > > indexes, replication, shard counts, etc. > > 7. In the case where the indexes never turn green, re-run the test a > > couple times > > 8. Allocated -Xmx1024m to the entire test case > > > > Assumptions: > > > > 1. A cluster should always strive toward GREEN when there is more than > > one node > > 2. If a node is lost, the cluster should always rebalance until it > > becomes GREEN again (assuming more than one node) > > > > Results: > > > > In many cases, ES fails to maintain appropriate balance. It has most > > success when there are 5 indexes, but still not perfect. As the > > number of nodes increase, the odds for failure increase on node > > failure. Also, as the number of indexes increase, the odds for > > failure increase. > > > > Test Class:[http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) > > Detailed Results: > > [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ)... > > > > The only dependencies I can see are the ones necessary to build > > elasticsearch. > > > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes to > > create] [nodata, somedata, or alldata] ---
**Author:** ![Stephen\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephen_2/32/3274_2.png) [@Stephen\_2](https://discuss.elastic.co/u/Stephen_2)\ **Post date:** [July 20, 2010, 11:02pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/9 "2010-07-20T23:02:51Z")
Awesome! Sorry I missed your message. I'll take a look at it. Glad I was able to be of assistance. I hope my continuous nagging isn't too much of a bother. 🙂 We're evaluating ES hard-core for use in a bunch of critical systems and can't wait for it to hit the maturity level where we can use it. I'm confident that your approach is the right one -- hope I can contribute if only with test cases. Regards, Stephen. On Jul 20, 8:48 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Maybe you missed my previous email from Jul 18, here it is again 🙂 : > > " > Hi, > > I just pushed a fix for this, I can't get it to happen with your test > anymore. If you can verify it, it would be great. I will write some built in > similar tests for this that will run as part of the test suite of > elasticsearch. > " > > -shay.banon > > On Tue, Jul 20, 2010 at 2:19 PM, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > ... is there a chance anybody's looked at this? > > > On Jul 17, 11:14 am, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > Okay, > > > > So I've spent all morning writing a fairly involved test class which > > > starts up its own nodes. I then ran a series of test cases that seem > > > to reproduce the health issue fairly regularly. The methodology I > > > used to execute the tests is as follows: > > > > 1. Vary the number of nodes from 2 to 4 > > > 2. Vary the number of indexes between 5 and 20 > > > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% > > > (alldata) > > > 4. When the number of nodes is greater than 2, and all indexes are > > > GREEN, and then kill the LAST node > > > 5. Before declaring the test to be a failure, wait for the log to stop > > > moving (after waiting 15-30 seconds) > > > 6. Used default values for everything -- no special configuration of > > > indexes, replication, shard counts, etc. > > > 7. In the case where the indexes never turn green, re-run the test a > > > couple times > > > 8. Allocated -Xmx1024m to the entire test case > > > > Assumptions: > > > > 1. A cluster should always strive toward GREEN when there is more than > > > one node > > > 2. If a node is lost, the cluster should always rebalance until it > > > becomes GREEN again (assuming more than one node) > > > > Results: > > > > In many cases, ES fails to maintain appropriate balance. It has most > > > success when there are 5 indexes, but still not perfect. As the > > > number of nodes increase, the odds for failure increase on node > > > failure. Also, as the number of indexes increase, the odds for > > > failure increase. > > > > Test Class:[http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) > > > Detailed Results: > > > [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ)... > > > > The only dependencies I can see are the ones necessary to build > > > elasticsearch. > > > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes to > > > create] [nodata, somedata, or alldata] ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 20, 2010, 11:04pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/10 "2010-07-20T23:04:30Z")
No bother at all, this is exactly the kind of things that will help elasticsearch mature. 0.9 is shaping up to be a really good release! -shay.banon On Wed, Jul 21, 2010 at 2:02 AM, Stephen [scstarkey@gmail.com](mailto:scstarkey@gmail.com) wrote: > Awesome! Sorry I missed your message. I'll take a look at it. Glad > I was able to be of assistance. > > I hope my continuous nagging isn't too much of a bother. 🙂 We're > evaluating ES hard-core for use in a bunch of critical systems and > can't wait for it to hit the maturity level where we can use it. I'm > confident that your approach is the right one -- hope I can contribute > if only with test cases. > > Regards, > > Stephen. > > On Jul 20, 8:48 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Maybe you missed my previous email from Jul 18, here it is again 🙂 : > > > > " > > Hi, > > > > I just pushed a fix for this, I can't get it to happen with your test > > anymore. If you can verify it, it would be great. I will write some built > > in > > similar tests for this that will run as part of the test suite of > > elasticsearch. > > " > > > > -shay.banon > > > > On Tue, Jul 20, 2010 at 2:19 PM, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > ... is there a chance anybody's looked at this? > > > > > On Jul 17, 11:14 am, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > > > Okay, > > > > > > So I've spent all morning writing a fairly involved test class which > > > > starts up its own nodes. I then ran a series of test cases that seem > > > > to reproduce the health issue fairly regularly. The methodology I > > > > used to execute the tests is as follows: > > > > > > 1. Vary the number of nodes from 2 to 4 > > > > 2. Vary the number of indexes between 5 and 20 > > > > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% > > > > (alldata) > > > > 4. When the number of nodes is greater than 2, and all indexes are > > > > GREEN, and then kill the LAST node > > > > 5. Before declaring the test to be a failure, wait for the log to > > > > stop > > > > moving (after waiting 15-30 seconds) > > > > 6. Used default values for everything -- no special configuration of > > > > indexes, replication, shard counts, etc. > > > > 7. In the case where the indexes never turn green, re-run the test a > > > > couple times > > > > 8. Allocated -Xmx1024m to the entire test case > > > > > > Assumptions: > > > > > > 1. A cluster should always strive toward GREEN when there is more > > > > than > > > > one node > > > > 2. If a node is lost, the cluster should always rebalance until it > > > > becomes GREEN again (assuming more than one node) > > > > > > Results: > > > > > > In many cases, ES fails to maintain appropriate balance. It has most > > > > success when there are 5 indexes, but still not perfect. As the > > > > number of nodes increase, the odds for failure increase on node > > > > failure. Also, as the number of indexes increase, the odds for > > > > failure increase. > > > > > > Test Class:[http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) > > > > Detailed Results: > > > > [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ). > > > > .. > > > > > > The only dependencies I can see are the ones necessary to build > > > > elasticsearch. > > > > > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes > > > > to > > > > create] [nodata, somedata, or alldata] ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 20, 2010, 11:05pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/11 "2010-07-20T23:05:12Z")
One more thing, as I want to release 0.9 soon (matter of days), it would really help if you can verify on your end that its fixed. If there are still problems, I would love to try and see if they can be tackled for 0.9. -shay.banon On Wed, Jul 21, 2010 at 2:04 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > No bother at all, this is exactly the kind of things that will help > elasticsearch mature. 0.9 is shaping up to be a really good release! > > -shay.banon > > On Wed, Jul 21, 2010 at 2:02 AM, Stephen [scstarkey@gmail.com](mailto:scstarkey@gmail.com) wrote: > > > Awesome! Sorry I missed your message. I'll take a look at it. Glad > > I was able to be of assistance. > > > > I hope my continuous nagging isn't too much of a bother. 🙂 We're > > evaluating ES hard-core for use in a bunch of critical systems and > > can't wait for it to hit the maturity level where we can use it. I'm > > confident that your approach is the right one -- hope I can contribute > > if only with test cases. > > > > Regards, > > > > Stephen. > > > > On Jul 20, 8:48 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Maybe you missed my previous email from Jul 18, here it is again 🙂 : > > > > > > " > > > Hi, > > > > > > I just pushed a fix for this, I can't get it to happen with your test > > > anymore. If you can verify it, it would be great. I will write some > > > built in > > > similar tests for this that will run as part of the test suite of > > > elasticsearch. > > > " > > > > > > -shay.banon > > > > > > On Tue, Jul 20, 2010 at 2:19 PM, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > > > ... is there a chance anybody's looked at this? > > > > > > > On Jul 17, 11:14 am, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > > > > > Okay, > > > > > > > > So I've spent all morning writing a fairly involved test class which > > > > > starts up its own nodes. I then ran a series of test cases that > > > > > seem > > > > > to reproduce the health issue fairly regularly. The methodology I > > > > > used to execute the tests is as follows: > > > > > > > > 1. Vary the number of nodes from 2 to 4 > > > > > 2. Vary the number of indexes between 5 and 20 > > > > > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% > > > > > (alldata) > > > > > 4. When the number of nodes is greater than 2, and all indexes are > > > > > GREEN, and then kill the LAST node > > > > > 5. Before declaring the test to be a failure, wait for the log to > > > > > stop > > > > > moving (after waiting 15-30 seconds) > > > > > 6. Used default values for everything -- no special configuration of > > > > > indexes, replication, shard counts, etc. > > > > > 7. In the case where the indexes never turn green, re-run the test a > > > > > couple times > > > > > 8. Allocated -Xmx1024m to the entire test case > > > > > > > > Assumptions: > > > > > > > > 1. A cluster should always strive toward GREEN when there is more > > > > > than > > > > > one node > > > > > 2. If a node is lost, the cluster should always rebalance until it > > > > > becomes GREEN again (assuming more than one node) > > > > > > > > Results: > > > > > > > > In many cases, ES fails to maintain appropriate balance. It has > > > > > most > > > > > success when there are 5 indexes, but still not perfect. As the > > > > > number of nodes increase, the odds for failure increase on node > > > > > failure. Also, as the number of indexes increase, the odds for > > > > > failure increase. > > > > > > > > Test Class:[http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) > > > > > Detailed Results: > > > > > [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ). > > > > > .. > > > > > > > > The only dependencies I can see are the ones necessary to build > > > > > elasticsearch. > > > > > > > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes > > > > > to > > > > > create] [nodata, somedata, or alldata] ---
**Author:** ![Stephen\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephen_2/32/3274_2.png) [@Stephen\_2](https://discuss.elastic.co/u/Stephen_2)\ **Post date:** [July 22, 2010, 11:58am UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/12 "2010-07-22T11:58:50Z")
Hey Shay, I just re-ran my automated tests and executed a bunch of manual tests and everything seems great on the replication and recovery front. I found a minor issue where documents get duplicated when recovered from the filesystem gateway, and will probably do some cross-computer testing as well (thus far have been focusing on same-machine stuff). Can't wait for the release! Stephen. On Jul 20, 6:05 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > One more thing, as I want to release 0.9 soon (matter of days), it would > really help if you can verify on your end that its fixed. If there are still > problems, I would love to try and see if they can be tackled for 0.9. > > -shay.banon > > On Wed, Jul 21, 2010 at 2:04 AM, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)wrote: > > > No bother at all, this is exactly the kind of things that will help > > elasticsearch mature. 0.9 is shaping up to be a really good release! > > > -shay.banon > > > On Wed, Jul 21, 2010 at 2:02 AM, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > Awesome! Sorry I missed your message. I'll take a look at it. Glad > > > I was able to be of assistance. > > > > I hope my continuous nagging isn't too much of a bother. 🙂 We're > > > evaluating ES hard-core for use in a bunch of critical systems and > > > can't wait for it to hit the maturity level where we can use it. I'm > > > confident that your approach is the right one -- hope I can contribute > > > if only with test cases. > > > > Regards, > > > > Stephen. > > > > On Jul 20, 8:48 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > Maybe you missed my previous email from Jul 18, here it is again 🙂 : > > > > > " > > > > Hi, > > > > > I just pushed a fix for this, I can't get it to happen with your test > > > > anymore. If you can verify it, it would be great. I will write some > > > > built in > > > > similar tests for this that will run as part of the test suite of > > > > elasticsearch. > > > > " > > > > > -shay.banon > > > > > On Tue, Jul 20, 2010 at 2:19 PM, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > > > > > ... is there a chance anybody's looked at this? > > > > > > On Jul 17, 11:14 am, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > > > > > > > Okay, > > > > > > > So I've spent all morning writing a fairly involved test class which > > > > > > starts up its own nodes. I then ran a series of test cases that > > > > > > seem > > > > > > to reproduce the health issue fairly regularly. The methodology I > > > > > > used to execute the tests is as follows: > > > > > > > 1. Vary the number of nodes from 2 to 4 > > > > > > 2. Vary the number of indexes between 5 and 20 > > > > > > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to 100% > > > > > > (alldata) > > > > > > 4. When the number of nodes is greater than 2, and all indexes are > > > > > > GREEN, and then kill the LAST node > > > > > > 5. Before declaring the test to be a failure, wait for the log to > > > > > > stop > > > > > > moving (after waiting 15-30 seconds) > > > > > > 6. Used default values for everything -- no special configuration of > > > > > > indexes, replication, shard counts, etc. > > > > > > 7. In the case where the indexes never turn green, re-run the test a > > > > > > couple times > > > > > > 8. Allocated -Xmx1024m to the entire test case > > > > > > > Assumptions: > > > > > > > 1. A cluster should always strive toward GREEN when there is more > > > > > > than > > > > > > one node > > > > > > 2. If a node is lost, the cluster should always rebalance until it > > > > > > becomes GREEN again (assuming more than one node) > > > > > > > Results: > > > > > > > In many cases, ES fails to maintain appropriate balance. It has > > > > > > most > > > > > > success when there are 5 indexes, but still not perfect. As the > > > > > > number of nodes increase, the odds for failure increase on node > > > > > > failure. Also, as the number of indexes increase, the odds for > > > > > > failure increase. > > > > > > > Test Class:[http://www.sourcepod.com/puyttw27-3473](http://www.sourcepod.com/puyttw27-3473) > > > > > > Detailed Results: > > > > > > [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ). > > > > > > .. > > > > > > > The only dependencies I can see are the ones necessary to build > > > > > > elasticsearch. > > > > > > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# indexes > > > > > > to > > > > > > create] [nodata, somedata, or alldata] ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 22, 2010, 6:06pm UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/13 "2010-07-22T18:06:24Z")
Hi, Great that this works. Are you sure that documents gets duplicate when recovery from gateway? If this happens, can you recreate it? -shay.banon On Thu, Jul 22, 2010 at 2:58 PM, Stephen [scstarkey@gmail.com](mailto:scstarkey@gmail.com) wrote: > Hey Shay, > > I just re-ran my automated tests and executed a bunch of manual tests > and everything seems great on the replication and recovery front. I > found a minor issue where documents get duplicated when recovered from > the filesystem gateway, and will probably do some cross-computer > testing as well (thus far have been focusing on same-machine stuff). > > Can't wait for the release! > > Stephen. > > On Jul 20, 6:05 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > One more thing, as I want to release 0.9 soon (matter of days), it would > > really help if you can verify on your end that its fixed. If there are > > still > > problems, I would love to try and see if they can be tackled for 0.9. > > > > -shay.banon > > > > On Wed, Jul 21, 2010 at 2:04 AM, Shay Banon \< > > [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com)\>wrote: > > > > > No bother at all, this is exactly the kind of things that will help > > > elasticsearch mature. 0.9 is shaping up to be a really good release! > > > > > -shay.banon > > > > > On Wed, Jul 21, 2010 at 2:02 AM, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > > Awesome! Sorry I missed your message. I'll take a look at it. Glad > > > > I was able to be of assistance. > > > > > > I hope my continuous nagging isn't too much of a bother. 🙂 We're > > > > evaluating ES hard-core for use in a bunch of critical systems and > > > > can't wait for it to hit the maturity level where we can use it. I'm > > > > confident that your approach is the right one -- hope I can contribute > > > > if only with test cases. > > > > > > Regards, > > > > > > Stephen. > > > > > > On Jul 20, 8:48 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > > > > Maybe you missed my previous email from Jul 18, here it is again 🙂 > > > > > : > > > > > > > " > > > > > Hi, > > > > > > > I just pushed a fix for this, I can't get it to happen with your > > > > > test > > > > > anymore. If you can verify it, it would be great. I will write some > > > > > built in > > > > > similar tests for this that will run as part of the test suite of > > > > > elasticsearch. > > > > > " > > > > > > > -shay.banon > > > > > > > On Tue, Jul 20, 2010 at 2:19 PM, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) > > > > > wrote: > > > > > > > > > > > ... is there a chance anybody's looked at this? > > > > > > > > On Jul 17, 11:14 am, Stephen [scstar...@gmail.com](mailto:scstar...@gmail.com) wrote: > > > > > > > > > > > > > Okay, > > > > > > > > > So I've spent all morning writing a fairly involved test class > > > > > > > which > > > > > > > starts up its own nodes. I then ran a series of test cases that > > > > > > > seem > > > > > > > to reproduce the health issue fairly regularly. The methodology > > > > > > > I > > > > > > > used to execute the tests is as follows: > > > > > > > > > 1. Vary the number of nodes from 2 to 4 > > > > > > > 2. Vary the number of indexes between 5 and 20 > > > > > > > 3. Vary the mount of data from 0% (nodata) to 25% (somedata) to > > > > > > > 100% > > > > > > > (alldata) > > > > > > > 4. When the number of nodes is greater than 2, and all indexes > > > > > > > are > > > > > > > GREEN, and then kill the LAST node > > > > > > > 5. Before declaring the test to be a failure, wait for the log > > > > > > > to > > > > > > > stop > > > > > > > moving (after waiting 15-30 seconds) > > > > > > > 6. Used default values for everything -- no special > > > > > > > configuration of > > > > > > > indexes, replication, shard counts, etc. > > > > > > > 7. In the case where the indexes never turn green, re-run the > > > > > > > test a > > > > > > > couple times > > > > > > > 8. Allocated -Xmx1024m to the entire test case > > > > > > > > > Assumptions: > > > > > > > > > 1. A cluster should always strive toward GREEN when there is > > > > > > > more > > > > > > > than > > > > > > > one node > > > > > > > 2. If a node is lost, the cluster should always rebalance until > > > > > > > it > > > > > > > becomes GREEN again (assuming more than one node) > > > > > > > > > Results: > > > > > > > > > In many cases, ES fails to maintain appropriate balance. It has > > > > > > > most > > > > > > > success when there are 5 indexes, but still not perfect. As the > > > > > > > number of nodes increase, the odds for failure increase on node > > > > > > > failure. Also, as the number of indexes increase, the odds for > > > > > > > failure increase. > > > > > > > > > Test Class:[Loading...](http://www.sourcepod.com/puyttw27-3473) > > > > > > > Detailed Results: > > [http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ](http://spreadsheets.google.com/ccc?key=0AoyOQtG1DAe4dDhrVXpxMkZOMkYwQ). > > > > > .. > > > > > > > > > The only dependencies I can see are the ones necessary to build > > > > > > > elasticsearch. > > > > > > > > > Usage: com.pegby.ElasticSearchTester [# nodes to start] [# > > > > > > > indexes > > > > > > > to > > > > > > > create] [nodata, somedata, or alldata] ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:21am UTC](https://discuss.elastic.co/t/cluster-health-issues-with-0-9-0-snapshot/3114/14 "2017-07-06T04:21:48Z")