Search fails with "java.io.IOException: listener timeout after waiting for [30000] ms"

Elastic Stack Elasticsearch
1

We have a 10-node cluster having just a few GBs of data. We have 5 indexes (total of 28 primary shards) and replication of 1 for each shard. Last night we deleted 2 indexes and re-ingested data to these indexes. After this, we are seeing following error when we try to read data from any of the indexes :

java.io.IOException: listener timeout after waiting for [30000] ms
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:617) ~[rest-5.2.2.jar:5.2.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:212) ~[rest-5.2.2.jar:5.2.2]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:184) ~[rest-5.2.2.jar:5.2.2]

I have searched this forum and SO for similar issues, but the suggestions are mainly for long running ingestion or bulk updates which is not applicable to our scenario since we observe this error during search. If I try to run the same query using curl command by logging in to one of the ES nodes, it runs fine without any errors and returns the results.

This is the code we use to create RestClient

public RestClient getElasticServerRestClient(String sourceESServerIp, int sourceESServerPort,
		String sourceESServerProtocol, final String userName, final String password,String resource) {
	try {
		HttpHost elasticCacheHost = new HttpHost(sourceESServerIp, sourceESServerPort, sourceESServerProtocol);
		RestClientBuilder restClientBuilder = RestClient.builder(elasticCacheHost);
		restClientBuilder.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
			@Override
			public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
				int maxConnTotal = cacheConfigurations.getMaxTotalConnections();
				if (maxConnTotal > 0) {
					httpClientBuilder.setMaxConnTotal(maxConnTotal);
				}
				int maxConnPerRoute = cacheConfigurations.getMaxConnectionsPerRoute();
				if (maxConnPerRoute > 0) {
					httpClientBuilder.setMaxConnPerRoute(maxConnPerRoute);
				}

				if (userName != null && !userName.trim().isEmpty() && password != null && !password.trim().isEmpty()) {
					final CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); 
					credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(userName, 
							password));
					httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
				}
				return httpClientBuilder;
			}
		});


		RestClient elasticServerRestClient = restClientBuilder.build();
		return elasticServerRestClient;
	}
	catch (Throwable t) {
		log.error("Error in getElasticServerRestClient", t);
		return null;
	}
}

The cluster state is green and no sign of any other issues from logs also. Please share any suggestions on how to identify why it is failing.

Thanks & Regards,
Shobhana

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.