# Race condition upon startup? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 1, 2011, 8:07pm UTC](https://discuss.elastic.co/t/race-condition-upon-startup/5020 "2011-08-01T20:07:47Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [August 1, 2011, 8:07pm UTC](https://discuss.elastic.co/t/race-condition-upon-startup/5020/1 "2011-08-01T20:07:47Z")
Hi all, I am getting intermittent failures about missing indices in my code which uses the elasticsearch Java API. I can easily reproduce a similar exception in a stand-alone elasticsearch instance using elasticsearch-head as follows: 1. Start up an elasticsearch node that has an existing index (i.e. run \/bin/elastcsearch -f) 2. Immediately go to elasticsearch-head and quickly hit "Connect" repeatedly until yellow status appears 3. The elasticsearch logs show an exception: I get a similar exception that fails my unit tests when I try to do index operations on a local node immediately after waiting for yellow status. The above stack trace is from 0.17.2, but I get the same behavior in 0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was using 0.15.2. Any ideas on what the problem may be or how to work around it? thanks, Curtis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 1, 2011, 8:23pm UTC](https://discuss.elastic.co/t/race-condition-upon-startup/5020/2 "2011-08-01T20:23:48Z")
The status failure on specific shards can happen, they are not propagated as a complete failure to the client using the API. The other one you specify, the one with the yellow status and failing to index, can you post a sample code for it? On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) wrote: > Hi all, > > I am getting intermittent failures about missing indices in my code which > uses the elasticsearch Java API. I can easily reproduce a similar exception > in a stand-alone elasticsearch instance using elasticsearch-head as follows: > > 1. Start up an elasticsearch node that has an existing index (i.e. run > \/bin/elastcsearch -f) > > 2. Immediately go to elasticsearch-head and quickly hit "Connect" > repeatedly until yellow status appears > > 3. The elasticsearch logs show an exception: > > I get a similar exception that fails my unit tests when I try to do index > operations on a local node immediately after waiting for yellow status. > > The above stack trace is from 0.17.2, but I get the same behavior in 0.16.2 > and 0.16.5. I'm pretty sure I never ran into this problem when I was using > 0.15.2. > > Any ideas on what the problem may be or how to work around it? > > thanks, > > Curtis ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [August 1, 2011, 8:40pm UTC](https://discuss.elastic.co/t/race-condition-upon-startup/5020/3 "2011-08-01T20:40:59Z")
Here's one example (this is using version 0.16.2): org.elasticsearch.indices.IndexMissingException: [test\_cache\_directory] missing at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:165) at org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:132) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:153) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:43) at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61) at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:47) at org.elasticsearch.client.node.NodeIndicesAdminClient.refresh(NodeIndicesAdminClient.java:195) ... The Java code leading up to it is this (attempt to create index, wait for yellow, refresh): ``` try { ``` client.admin().indices().create(createIndexRequest(indexName).settings(settings)).actionGet(); } catch (IndexAlreadyExistsException e) { logIndexExists(e, indexName); } catch (RemoteTransportException r) { Throwable t = r.getMostSpecificCause(); if (t instanceof IndexAlreadyExistsException) { logIndexExists(t, indexName); } else { throw r; } } client.admin().cluster().health(clusterHealthRequest(indexName).waitForYellowStatus()).actionGet(); client.admin().indices().refresh(refreshRequest(indexName)).actionGet(); I can try to extract a stand-alone example if you need it. Curtis On Mon, Aug 1, 2011 at 1:23 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > The status failure on specific shards can happen, they are not propagated > as a complete failure to the client using the API. > > The other one you specify, the one with the yellow status and failing to > index, can you post a sample code for it? > > On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > Hi all, > > > > I am getting intermittent failures about missing indices in my code which > > uses the elasticsearch Java API. I can easily reproduce a similar exception > > in a stand-alone elasticsearch instance using elasticsearch-head as follows: > > > > 1. Start up an elasticsearch node that has an existing index (i.e. run > > \/bin/elastcsearch -f) > > > > 2. Immediately go to elasticsearch-head and quickly hit "Connect" > > repeatedly until yellow status appears > > > > 3. The elasticsearch logs show an exception: > > > > I get a similar exception that fails my unit tests when I try to do index > > operations on a local node immediately after waiting for yellow status. > > > > The above stack trace is from 0.17.2, but I get the same behavior in > > 0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was > > using 0.15.2. > > > > Any ideas on what the problem may be or how to work around it? > > > > thanks, > > > > Curtis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [August 1, 2011, 8:51pm UTC](https://discuss.elastic.co/t/race-condition-upon-startup/5020/4 "2011-08-01T20:51:27Z")
Strange, can you gist a simple recreation? Btw, there is an index exists API in 0.17. On Mon, Aug 1, 2011 at 11:40 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) wrote: > Here's one example (this is using version 0.16.2): > > org.elasticsearch.indices.IndexMissingException: [test\_cache\_directory] > missing > at > org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:165) > at > org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:132) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:153) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:43) > at > org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61) > at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:47) > at > org.elasticsearch.client.node.NodeIndicesAdminClient.refresh(NodeIndicesAdminClient.java:195) > ... > > The Java code leading up to it is this (attempt to create index, wait for > yellow, refresh): > > ``` > try { > > ``` > > client.admin().indices().create(createIndexRequest(indexName).settings(settings)).actionGet(); > } catch (IndexAlreadyExistsException e) { > logIndexExists(e, indexName); > } catch (RemoteTransportException r) { > Throwable t = r.getMostSpecificCause(); > if (t instanceof IndexAlreadyExistsException) { > logIndexExists(t, indexName); > } else { > throw r; > } > } > > client.admin().cluster().health(clusterHealthRequest(indexName).waitForYellowStatus()).actionGet(); > > client.admin().indices().refresh(refreshRequest(indexName)).actionGet(); > > I can try to extract a stand-alone example if you need it. > > Curtis > > On Mon, Aug 1, 2011 at 1:23 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > The status failure on specific shards can happen, they are not propagated > > as a complete failure to the client using the API. > > > > The other one you specify, the one with the yellow status and failing to > > index, can you post a sample code for it? > > > > On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > Hi all, > > > > > > I am getting intermittent failures about missing indices in my code which > > > uses the elasticsearch Java API. I can easily reproduce a similar exception > > > in a stand-alone elasticsearch instance using elasticsearch-head as follows: > > > > > > 1. Start up an elasticsearch node that has an existing index (i.e. run > > > \/bin/elastcsearch -f) > > > > > > 2. Immediately go to elasticsearch-head and quickly hit "Connect" > > > repeatedly until yellow status appears > > > > > > 3. The elasticsearch logs show an exception: > > > > > > I get a similar exception that fails my unit tests when I try to do index > > > operations on a local node immediately after waiting for yellow status. > > > > > > The above stack trace is from 0.17.2, but I get the same behavior in > > > 0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was > > > using 0.15.2. > > > > > > Any ideas on what the problem may be or how to work around it? > > > > > > thanks, > > > > > > Curtis ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [August 1, 2011, 8:56pm UTC](https://discuss.elastic.co/t/race-condition-upon-startup/5020/5 "2011-08-01T20:56:45Z")
After a few more attempts, I can also reproduce the IndexMissingException in the stand-alone instance / elasticsearch-head case (in 0.16.2 but not 0.17.2): [2011-08-01 13:43:13,662][INFO][http] [Doctor Minerva‎] bound\_address {inet[/0.0.0.0:9200]}, publish\_address {inet[/ 192.168.27.116:9200]} [2011-08-01 13:43:13,664][INFO][node] [Doctor Minerva‎] {elasticsearch/0.16.2}[26119]: started [2011-08-01 13:43:14,048][DEBUG][action.admin.indices.status] [Doctor Minerva‎] [test][0], node[97SntgvNQuqXX2cToI7ZEQ], [P], s[INITIALIZING]: Failed to execute [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@6128453c ] org.elasticsearch.indices.IndexMissingException: [test] missing at org.elasticsearch.indices.InternalIndicesService.indexServiceSafe(InternalIndicesService.java:208) at org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:149) at org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:59) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:238) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.access$200(TransportBroadcastOperationAction.java:126) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:195) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:680) On Mon, Aug 1, 2011 at 1:40 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) wrote: > Here's one example (this is using version 0.16.2): > > org.elasticsearch.indices.IndexMissingException: [test\_cache\_directory] > missing > at > org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:165) > at > org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:132) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:153) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:43) > at > org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61) > at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:47) > at > org.elasticsearch.client.node.NodeIndicesAdminClient.refresh(NodeIndicesAdminClient.java:195) > ... > > The Java code leading up to it is this (attempt to create index, wait for > yellow, refresh): > > ``` > try { > > ``` > > client.admin().indices().create(createIndexRequest(indexName).settings(settings)).actionGet(); > } catch (IndexAlreadyExistsException e) { > logIndexExists(e, indexName); > } catch (RemoteTransportException r) { > Throwable t = r.getMostSpecificCause(); > if (t instanceof IndexAlreadyExistsException) { > logIndexExists(t, indexName); > } else { > throw r; > } > } > > client.admin().cluster().health(clusterHealthRequest(indexName).waitForYellowStatus()).actionGet(); > > client.admin().indices().refresh(refreshRequest(indexName)).actionGet(); > > I can try to extract a stand-alone example if you need it. > > Curtis > > On Mon, Aug 1, 2011 at 1:23 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > The status failure on specific shards can happen, they are not propagated > > as a complete failure to the client using the API. > > > > The other one you specify, the one with the yellow status and failing to > > index, can you post a sample code for it? > > > > On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > Hi all, > > > > > > I am getting intermittent failures about missing indices in my code which > > > uses the elasticsearch Java API. I can easily reproduce a similar exception > > > in a stand-alone elasticsearch instance using elasticsearch-head as follows: > > > > > > 1. Start up an elasticsearch node that has an existing index (i.e. run > > > \/bin/elastcsearch -f) > > > > > > 2. Immediately go to elasticsearch-head and quickly hit "Connect" > > > repeatedly until yellow status appears > > > > > > 3. The elasticsearch logs show an exception: > > > > > > I get a similar exception that fails my unit tests when I try to do index > > > operations on a local node immediately after waiting for yellow status. > > > > > > The above stack trace is from 0.17.2, but I get the same behavior in > > > 0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was > > > using 0.15.2. > > > > > > Any ideas on what the problem may be or how to work around it? > > > > > > thanks, > > > > > > Curtis ---
**Author:** ![Curtis\_Caravone](https://avatars.discourse-cdn.com/v4/letter/c/e47c2d/32.png) [@Curtis\_Caravone](https://discuss.elastic.co/u/Curtis_Caravone)\ **Post date:** [August 1, 2011, 11:51pm UTC](https://discuss.elastic.co/t/race-condition-upon-startup/5020/6 "2011-08-01T23:51:11Z")
I'm having trouble reproducing this in a small example, so this is all I have right now. I'll post an example if I can manage to reproduce it on a smaller scale. thanks, Curtis On Mon, Aug 1, 2011 at 1:40 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com) wrote: > Here's one example (this is using version 0.16.2): > > org.elasticsearch.indices.IndexMissingException: [test\_cache\_directory] > missing > at > org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:165) > at > org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:132) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:153) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71) > at > org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:43) > at > org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61) > at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:47) > at > org.elasticsearch.client.node.NodeIndicesAdminClient.refresh(NodeIndicesAdminClient.java:195) > ... > > The Java code leading up to it is this (attempt to create index, wait for > yellow, refresh): > > ``` > try { > > ``` > > client.admin().indices().create(createIndexRequest(indexName).settings(settings)).actionGet(); > } catch (IndexAlreadyExistsException e) { > logIndexExists(e, indexName); > } catch (RemoteTransportException r) { > Throwable t = r.getMostSpecificCause(); > if (t instanceof IndexAlreadyExistsException) { > logIndexExists(t, indexName); > } else { > throw r; > } > } > > client.admin().cluster().health(clusterHealthRequest(indexName).waitForYellowStatus()).actionGet(); > > client.admin().indices().refresh(refreshRequest(indexName)).actionGet(); > > I can try to extract a stand-alone example if you need it. > > Curtis > > On Mon, Aug 1, 2011 at 1:23 PM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > The status failure on specific shards can happen, they are not propagated > > as a complete failure to the client using the API. > > > > The other one you specify, the one with the yellow status and failing to > > index, can you post a sample code for it? > > > > On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone [caravone@gmail.com](mailto:caravone@gmail.com)wrote: > > > > > Hi all, > > > > > > I am getting intermittent failures about missing indices in my code which > > > uses the elasticsearch Java API. I can easily reproduce a similar exception > > > in a stand-alone elasticsearch instance using elasticsearch-head as follows: > > > > > > 1. Start up an elasticsearch node that has an existing index (i.e. run > > > \/bin/elastcsearch -f) > > > > > > 2. Immediately go to elasticsearch-head and quickly hit "Connect" > > > repeatedly until yellow status appears > > > > > > 3. The elasticsearch logs show an exception: > > > > > > I get a similar exception that fails my unit tests when I try to do index > > > operations on a local node immediately after waiting for yellow status. > > > > > > The above stack trace is from 0.17.2, but I get the same behavior in > > > 0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was > > > using 0.15.2. > > > > > > Any ideas on what the problem may be or how to work around it? > > > > > > thanks, > > > > > > Curtis ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:58am UTC](https://discuss.elastic.co/t/race-condition-upon-startup/5020/7 "2017-07-06T03:58:33Z")