I am currently using 3 data nodes nodes ES 6.1 cluster on PROD env.For the past week we have start getting below error
"java.lang.IllegalStateException: Request cannot be executed; I/O reactor status: STOPPED".
without anything going wrong in the cluster or in the nodes(as seen from the logs).Some other people have also posted same issue with a different resolution for all of them.So I want to know the concrete root cause why this is happening.As per my understanding the client gets into "CLOSED" status and any request after that starts failing.Also this is not happening for all the requests.So that suggests that its a node level issue.My code for connecting to ES:
for (String elasticHost : elasticSearchHosts){
httpHosts.add(new HttpHost(elasticHost, Integer.parseInt(elasticsearchPort)));
}
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(elasticUser, elasticPassword));
restHighLevelClient= new RestHighLevelClient(RestClient.builder(httpHosts.toArray(new HttpHost[httpHosts.size()])).setHttpClientConfigCallback(new HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder)
{ return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
}}));
return restHighLevelClient;
Any update would be helpful.