The issue were seeing on production seems to be that a shard of ES is stuck in the recovery state and we attempt to search it when it's in this state causing the exception.
From time to time we get this error, which results in nothing returning from a search, we are using ES version 1.7.1 on Found:
17:55:57 DEBUG action.search.type i17@z0 [2015-08-03 21:55:57,142][DEBUG][action.search.type ] All shards failed for phase: [dfs] org.elasticsearch.transport.RemoteTransportException: [instance-0000000018][inet[ ... ]][indices:data/read/search[phase/dfs]] Caused by: org.elasticsearch.index.shard.IllegalIndexShardStateException: [clamour_postv1][1] CurrentState[RECOVERING] operations only allowed when started/relocated at org.elasticsearch.index.shard.IndexShard.readAllowed(IndexShard.java:1004) at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:797) at org.elasticsearch.index.shard.IndexShard.acquireSearcher(IndexShard.java:793) at org.elasticsearch.search.SearchService.createContext(SearchService.java:564) at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544) at org.elasticsearch.search.SearchService.executeDfsPhase(SearchService.java:223) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchDfsTransportHandler.messageReceived(SearchServiceTransportAction.java:757) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchDfsTransportHandler.messageReceived(SearchServiceTransportAction.java:748) at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:176) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:279) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:36) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)