Race condition upon startup?

Elastic Stack Elasticsearch
1

Hi all,

I am getting intermittent failures about missing indices in my code which
uses the elasticsearch Java API. I can easily reproduce a similar exception
in a stand-alone elasticsearch instance using elasticsearch-head as follows:

  1. Start up an elasticsearch node that has an existing index (i.e. run
    <elasticsearch_home>/bin/elastcsearch -f)

  2. Immediately go to elasticsearch-head and quickly hit "Connect"
    repeatedly until yellow status appears

  3. The elasticsearch logs show an exception:

    [2011-08-01 12:49:07,217][DEBUG][action.admin.indices.status] [Armor]
    [test][3], node[y5QCD5imRzmfbUZcIhkMsQ], [P], s[INITIALIZING]: Failed to
    execute
    [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@2f6e4ddd
    ]
    org.elasticsearch.index.IndexShardMissingException: [test][3] missing
    at
    org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:172)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:135)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:58)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:227)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:205)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:181)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

I get a similar exception that fails my unit tests when I try to do index
operations on a local node immediately after waiting for yellow status.

The above stack trace is from 0.17.2, but I get the same behavior in 0.16.2
and 0.16.5. I'm pretty sure I never ran into this problem when I was using
0.15.2.

Any ideas on what the problem may be or how to work around it?

thanks,

Curtis

2

The status failure on specific shards can happen, they are not propagated as
a complete failure to the client using the API.

The other one you specify, the one with the yellow status and failing to
index, can you post a sample code for it?

On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone caravone@gmail.com wrote:

Hi all,

I am getting intermittent failures about missing indices in my code which
uses the elasticsearch Java API. I can easily reproduce a similar exception
in a stand-alone elasticsearch instance using elasticsearch-head as follows:

  1. Start up an elasticsearch node that has an existing index (i.e. run
    <elasticsearch_home>/bin/elastcsearch -f)

  2. Immediately go to elasticsearch-head and quickly hit "Connect"
    repeatedly until yellow status appears

  3. The elasticsearch logs show an exception:

    [2011-08-01 12:49:07,217][DEBUG][action.admin.indices.status] [Armor]
    [test][3], node[y5QCD5imRzmfbUZcIhkMsQ], [P], s[INITIALIZING]: Failed to
    execute
    [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@2f6e4ddd
    ]
    org.elasticsearch.index.IndexShardMissingException: [test][3] missing
    at
    org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:172)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:135)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:58)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:227)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:205)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:181)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

I get a similar exception that fails my unit tests when I try to do index
operations on a local node immediately after waiting for yellow status.

The above stack trace is from 0.17.2, but I get the same behavior in 0.16.2
and 0.16.5. I'm pretty sure I never ran into this problem when I was using
0.15.2.

Any ideas on what the problem may be or how to work around it?

thanks,

Curtis

3

Here's one example (this is using version 0.16.2):

org.elasticsearch.indices.IndexMissingException: [test_cache_directory]
missing
at
org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:165)
at
org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:132)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:153)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:43)
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61)
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:47)
at
org.elasticsearch.client.node.NodeIndicesAdminClient.refresh(NodeIndicesAdminClient.java:195)
...

The Java code leading up to it is this (attempt to create index, wait for
yellow, refresh):

    try {

client.admin().indices().create(createIndexRequest(indexName).settings(settings)).actionGet();
} catch (IndexAlreadyExistsException e) {
logIndexExists(e, indexName);
} catch (RemoteTransportException r) {
Throwable t = r.getMostSpecificCause();
if (t instanceof IndexAlreadyExistsException) {
logIndexExists(t, indexName);
} else {
throw r;
}
}

client.admin().cluster().health(clusterHealthRequest(indexName).waitForYellowStatus()).actionGet();

client.admin().indices().refresh(refreshRequest(indexName)).actionGet();

I can try to extract a stand-alone example if you need it.

Curtis

On Mon, Aug 1, 2011 at 1:23 PM, Shay Banon kimchy@gmail.com wrote:

The status failure on specific shards can happen, they are not propagated
as a complete failure to the client using the API.

The other one you specify, the one with the yellow status and failing to
index, can you post a sample code for it?

On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone caravone@gmail.comwrote:

Hi all,

I am getting intermittent failures about missing indices in my code which
uses the elasticsearch Java API. I can easily reproduce a similar exception
in a stand-alone elasticsearch instance using elasticsearch-head as follows:

  1. Start up an elasticsearch node that has an existing index (i.e. run
    <elasticsearch_home>/bin/elastcsearch -f)

  2. Immediately go to elasticsearch-head and quickly hit "Connect"
    repeatedly until yellow status appears

  3. The elasticsearch logs show an exception:

    [2011-08-01 12:49:07,217][DEBUG][action.admin.indices.status] [Armor]
    [test][3], node[y5QCD5imRzmfbUZcIhkMsQ], [P], s[INITIALIZING]: Failed to
    execute
    [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@2f6e4ddd
    ]
    org.elasticsearch.index.IndexShardMissingException: [test][3] missing
    at
    org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:172)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:135)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:58)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:227)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:205)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:181)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

I get a similar exception that fails my unit tests when I try to do index
operations on a local node immediately after waiting for yellow status.

The above stack trace is from 0.17.2, but I get the same behavior in
0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was
using 0.15.2.

Any ideas on what the problem may be or how to work around it?

thanks,

Curtis

4

Strange, can you gist a simple recreation? Btw, there is an index exists API
in 0.17.

On Mon, Aug 1, 2011 at 11:40 PM, Curtis Caravone caravone@gmail.com wrote:

Here's one example (this is using version 0.16.2):

org.elasticsearch.indices.IndexMissingException: [test_cache_directory]
missing
at
org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:165)
at
org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:132)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:153)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:43)
at
org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61)
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:47)
at
org.elasticsearch.client.node.NodeIndicesAdminClient.refresh(NodeIndicesAdminClient.java:195)
...

The Java code leading up to it is this (attempt to create index, wait for
yellow, refresh):

    try {

client.admin().indices().create(createIndexRequest(indexName).settings(settings)).actionGet();
} catch (IndexAlreadyExistsException e) {
logIndexExists(e, indexName);
} catch (RemoteTransportException r) {
Throwable t = r.getMostSpecificCause();
if (t instanceof IndexAlreadyExistsException) {
logIndexExists(t, indexName);
} else {
throw r;
}
}

client.admin().cluster().health(clusterHealthRequest(indexName).waitForYellowStatus()).actionGet();

client.admin().indices().refresh(refreshRequest(indexName)).actionGet();

I can try to extract a stand-alone example if you need it.

Curtis

On Mon, Aug 1, 2011 at 1:23 PM, Shay Banon kimchy@gmail.com wrote:

The status failure on specific shards can happen, they are not propagated
as a complete failure to the client using the API.

The other one you specify, the one with the yellow status and failing to
index, can you post a sample code for it?

On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone caravone@gmail.comwrote:

Hi all,

I am getting intermittent failures about missing indices in my code which
uses the elasticsearch Java API. I can easily reproduce a similar exception
in a stand-alone elasticsearch instance using elasticsearch-head as follows:

  1. Start up an elasticsearch node that has an existing index (i.e. run
    <elasticsearch_home>/bin/elastcsearch -f)

  2. Immediately go to elasticsearch-head and quickly hit "Connect"
    repeatedly until yellow status appears

  3. The elasticsearch logs show an exception:

    [2011-08-01 12:49:07,217][DEBUG][action.admin.indices.status] [Armor]
    [test][3], node[y5QCD5imRzmfbUZcIhkMsQ], [P], s[INITIALIZING]: Failed to
    execute
    [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@2f6e4ddd
    ]
    org.elasticsearch.index.IndexShardMissingException: [test][3] missing
    at
    org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:172)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:135)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:58)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:227)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:205)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:181)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

I get a similar exception that fails my unit tests when I try to do index
operations on a local node immediately after waiting for yellow status.

The above stack trace is from 0.17.2, but I get the same behavior in
0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was
using 0.15.2.

Any ideas on what the problem may be or how to work around it?

thanks,

Curtis

5

After a few more attempts, I can also reproduce the IndexMissingException in
the stand-alone instance / elasticsearch-head case (in 0.16.2 but not
0.17.2):

[2011-08-01 13:43:13,662][INFO ][http ] [Doctor
Minerva‎] bound_address {inet[/0.0.0.0:9200]}, publish_address {inet[/
192.168.27.116:9200]}
[2011-08-01 13:43:13,664][INFO ][node ] [Doctor
Minerva‎] {elasticsearch/0.16.2}[26119]: started
[2011-08-01 13:43:14,048][DEBUG][action.admin.indices.status] [Doctor
Minerva‎] [test][0], node[97SntgvNQuqXX2cToI7ZEQ], [P], s[INITIALIZING]:
Failed to execute
[org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@6128453c
]
org.elasticsearch.indices.IndexMissingException: [test] missing
at
org.elasticsearch.indices.InternalIndicesService.indexServiceSafe(InternalIndicesService.java:208)
at
org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:149)
at
org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:59)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:238)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.access$200(TransportBroadcastOperationAction.java:126)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:195)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)

On Mon, Aug 1, 2011 at 1:40 PM, Curtis Caravone caravone@gmail.com wrote:

Here's one example (this is using version 0.16.2):

org.elasticsearch.indices.IndexMissingException: [test_cache_directory]
missing
at
org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:165)
at
org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:132)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:153)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:43)
at
org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61)
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:47)
at
org.elasticsearch.client.node.NodeIndicesAdminClient.refresh(NodeIndicesAdminClient.java:195)
...

The Java code leading up to it is this (attempt to create index, wait for
yellow, refresh):

    try {

client.admin().indices().create(createIndexRequest(indexName).settings(settings)).actionGet();
} catch (IndexAlreadyExistsException e) {
logIndexExists(e, indexName);
} catch (RemoteTransportException r) {
Throwable t = r.getMostSpecificCause();
if (t instanceof IndexAlreadyExistsException) {
logIndexExists(t, indexName);
} else {
throw r;
}
}

client.admin().cluster().health(clusterHealthRequest(indexName).waitForYellowStatus()).actionGet();

client.admin().indices().refresh(refreshRequest(indexName)).actionGet();

I can try to extract a stand-alone example if you need it.

Curtis

On Mon, Aug 1, 2011 at 1:23 PM, Shay Banon kimchy@gmail.com wrote:

The status failure on specific shards can happen, they are not propagated
as a complete failure to the client using the API.

The other one you specify, the one with the yellow status and failing to
index, can you post a sample code for it?

On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone caravone@gmail.comwrote:

Hi all,

I am getting intermittent failures about missing indices in my code which
uses the elasticsearch Java API. I can easily reproduce a similar exception
in a stand-alone elasticsearch instance using elasticsearch-head as follows:

  1. Start up an elasticsearch node that has an existing index (i.e. run
    <elasticsearch_home>/bin/elastcsearch -f)

  2. Immediately go to elasticsearch-head and quickly hit "Connect"
    repeatedly until yellow status appears

  3. The elasticsearch logs show an exception:

    [2011-08-01 12:49:07,217][DEBUG][action.admin.indices.status] [Armor]
    [test][3], node[y5QCD5imRzmfbUZcIhkMsQ], [P], s[INITIALIZING]: Failed to
    execute
    [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@2f6e4ddd
    ]
    org.elasticsearch.index.IndexShardMissingException: [test][3] missing
    at
    org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:172)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:135)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:58)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:227)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:205)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:181)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

I get a similar exception that fails my unit tests when I try to do index
operations on a local node immediately after waiting for yellow status.

The above stack trace is from 0.17.2, but I get the same behavior in
0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was
using 0.15.2.

Any ideas on what the problem may be or how to work around it?

thanks,

Curtis

6

I'm having trouble reproducing this in a small example, so this is all I
have right now. I'll post an example if I can manage to reproduce it on a
smaller scale.

thanks,

Curtis

On Mon, Aug 1, 2011 at 1:40 PM, Curtis Caravone caravone@gmail.com wrote:

Here's one example (this is using version 0.16.2):

org.elasticsearch.indices.IndexMissingException: [test_cache_directory]
missing
at
org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:165)
at
org.elasticsearch.cluster.metadata.MetaData.concreteIndices(MetaData.java:132)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.(TransportBroadcastOperationAction.java:153)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71)
at
org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:43)
at
org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:61)
at org.elasticsearch.action.support.BaseAction.execute(BaseAction.java:47)
at
org.elasticsearch.client.node.NodeIndicesAdminClient.refresh(NodeIndicesAdminClient.java:195)
...

The Java code leading up to it is this (attempt to create index, wait for
yellow, refresh):

    try {

client.admin().indices().create(createIndexRequest(indexName).settings(settings)).actionGet();
} catch (IndexAlreadyExistsException e) {
logIndexExists(e, indexName);
} catch (RemoteTransportException r) {
Throwable t = r.getMostSpecificCause();
if (t instanceof IndexAlreadyExistsException) {
logIndexExists(t, indexName);
} else {
throw r;
}
}

client.admin().cluster().health(clusterHealthRequest(indexName).waitForYellowStatus()).actionGet();

client.admin().indices().refresh(refreshRequest(indexName)).actionGet();

I can try to extract a stand-alone example if you need it.

Curtis

On Mon, Aug 1, 2011 at 1:23 PM, Shay Banon kimchy@gmail.com wrote:

The status failure on specific shards can happen, they are not propagated
as a complete failure to the client using the API.

The other one you specify, the one with the yellow status and failing to
index, can you post a sample code for it?

On Mon, Aug 1, 2011 at 11:07 PM, Curtis Caravone caravone@gmail.comwrote:

Hi all,

I am getting intermittent failures about missing indices in my code which
uses the elasticsearch Java API. I can easily reproduce a similar exception
in a stand-alone elasticsearch instance using elasticsearch-head as follows:

  1. Start up an elasticsearch node that has an existing index (i.e. run
    <elasticsearch_home>/bin/elastcsearch -f)

  2. Immediately go to elasticsearch-head and quickly hit "Connect"
    repeatedly until yellow status appears

  3. The elasticsearch logs show an exception:

    [2011-08-01 12:49:07,217][DEBUG][action.admin.indices.status] [Armor]
    [test][3], node[y5QCD5imRzmfbUZcIhkMsQ], [P], s[INITIALIZING]: Failed to
    execute
    [org.elasticsearch.action.admin.indices.status.IndicesStatusRequest@2f6e4ddd
    ]
    org.elasticsearch.index.IndexShardMissingException: [test][3] missing
    at
    org.elasticsearch.index.service.InternalIndexService.shardSafe(InternalIndexService.java:172)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:135)
    at
    org.elasticsearch.action.admin.indices.status.TransportIndicesStatusAction.shardOperation(TransportIndicesStatusAction.java:58)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:227)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:205)
    at
    org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction$1.run(TransportBroadcastOperationAction.java:181)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at
    java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:680)

I get a similar exception that fails my unit tests when I try to do index
operations on a local node immediately after waiting for yellow status.

The above stack trace is from 0.17.2, but I get the same behavior in
0.16.2 and 0.16.5. I'm pretty sure I never ran into this problem when I was
using 0.15.2.

Any ideas on what the problem may be or how to work around it?

thanks,

Curtis