Dear all,
For the past several days i am trying to solve a problem with delaying of requests for search in Elastic Search (And actually this is what our customers are reporting - that sometimes when they do a search there is a delay before returning the results). We are using Elastic Search Cloud Service (SaaS) and our java application (which currently is hosted on Azure Cloud) is connecting to it with ElasticSearch RestHighLevelClient. I did some investigation and seems that if for a long period of time nobody do a search (lets say 1-2 hours) then the first search after that is kind of slow - 3-4 sec. After this first search everything seems to be ok, and every next search is returning results almost immediately (i can hit one and the same search 20 times and it responds almost immediately). Just for your information the search which we are doing is pretty simple, nothing complex.
Here is some information about the current state of our code implementation:
We are initializing the RestHighLevelClient at the very beginning when the app is started and we are using this client for every single interaction with ES Cloud.
@Bean
RestHighLevelClient restHighLevelClient() {
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(username, password));
String esHost = "our.es.host.in.es.cloud";
int esHostPort = 9243;
return new RestHighLevelClient(
RestClient.builder(new HttpHost(esHost, esHostPort, "https"))
.setHttpClientConfigCallback(new RestClientBuilder.HttpClientConfigCallback() {
@Override
public HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder) {
httpClientBuilder.disableAuthCaching();
return httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider);
}
}));
}
Searching for some results:
@Autowired
private final RestHighLevelClient esClient;
.....
.....
SearchResponse searchResponse = esClient.search(searchRequest, RequestOptions.DEFAULT);
So...what i did as a test was to add esClient.ping() which will be executed every 35 minutes in order to check the connection to ES. I was surprised that on every 3-4 ping() requests one is unsuccessful, here is the error :
22:00:00.409 [pool-6-thread-1] INFO c.b.t.s.ElasticSearchProductDataService - Pinging the remote Elasticsearch cluster: true
22:35:00.379 [pool-6-thread-1] INFO c.b.t.s.ElasticSearchProductDataService - Pinging the remote Elasticsearch cluster: true
23:00:00.396 [pool-6-thread-1] INFO c.b.t.s.ElasticSearchProductDataService - Pinging the remote Elasticsearch cluster: true
23:35:30.029 [pool-6-thread-1] ERROR o.s.s.s.TaskUtils$LoggingErrorHandler - Unexpected error occurred in scheduled task.
java.net.SocketTimeoutException: null
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:944)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734)
at org.elasticsearch.client.RestHighLevelClient.ping(RestHighLevelClient.java:694)
at com.app.appi.service.ElasticSearchProductDataService.checkConnection(ElasticSearchProductDataService.java:430)
at sun.reflect.GeneratedMethodAccessor136.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:81)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketTimeoutException: null
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.timeout(HttpAsyncRequestExecutor.java:375)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:92)
at org.apache.http.impl.nio.client.InternalIODispatch.onTimeout(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.timeout(AbstractIODispatch.java:175)
at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionTimedOut(BaseIOReactor.java:263)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.timeoutCheck(AbstractIOReactor.java:492)
at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:213)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
... 1 common frames omitted
00:00:00.381 [pool-6-thread-1] INFO c.b.t.s.ElasticSearchProductDataService - Pinging the remote Elasticsearch cluster: true
As you can see from the logs the next ping() request after this exception is successful. So i guess at some point of the time the RestHighlevelClient is loosing the connection to ES and it takes some time to connect again..which probably is resulting as a delay for the users search.
Any ideas how to solve this problem?
Thank you!