# Intermittent errors connecting to ES cluster when using High Level Rest client **URL:** \ **Category:** Elasticsearch\ **Tags:** language-clients\ **Created:** [September 8, 2020, 10:15pm UTC](https://discuss.elastic.co/t/intermittent-errors-connecting-to-es-cluster-when-using-high-level-rest-client/247929 "2020-09-08T22:15:30Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![jdevilla20](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jdevilla20/32/75281_2.png) [@jdevilla20](https://discuss.elastic.co/u/jdevilla20)\ **Post date:** [September 8, 2020, 10:15pm UTC](https://discuss.elastic.co/t/intermittent-errors-connecting-to-es-cluster-when-using-high-level-rest-client/247929/1 "2020-09-08T22:15:30Z")
**Elasticsearch version** ( `bin/elasticsearch --version` ): 7.6.2 **Plugins installed** : **JVM version** ( `java -version` ): sh-4.2# /usr/share/elasticsearch/jdk/bin/java -version openjdk version "13.0.2" 2020-01-14 OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.2+8) OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.2+8, mixed mode, sharing) **OS version** ( `uname -a` if on a Unix-like system): Linux x86\_64 x86\_64 x86\_64 GNU/Linux **Description of the problem including expected versus actual behavior** : **Steps to reproduce** : I am running a 3 node ES cluster on Kubernetes and performing about 80 upserts per minute. When connecting via the High Level Rest client, I see intermittent errors (see below). Other times, the updates work fine. When I look at my CPU usage, the hot cpu is running at 15% usage. I am not sure if the errors are with the cluster or with the Rest client. **Rest Client initialization** : ```auto private RestHighLevelClient initClient(ElasticSearchStoreConfig config) throws IOException { String password = "foobar"; final CredentialsProvider credentialsProvider = new BasicCredentialsProvider(); credentialsProvider.setCredentials(AuthScope.ANY, new UsernamePasswordCredentials(config.getUserName(), password)); RestClientBuilder builder = RestClient.builder(new HttpHost(config.getHostname(), config.getPort(), config.getElasticSearchScheme())) .setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)); return new RestHighLevelClient(builder); } ``` I am running the update as follows (there is a retry policy): ```auto UpdateRequest updateRequest = mapper.update(key, data); UpdateResponse response = esClient.update(updateRequest, RequestOptions.DEFAULT); ``` **Provide logs (if relevant)** : ```auto [DEBUG] 2020-09-02 17:57:49.351 [pool-7-thread-1] PoolingNHttpClientConnectionManager: - Connection request failed java.net.ConnectException: Connection refused at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?] at sun.nio.ch.Net.pollConnectNow(Net.java:579) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820) ~[?:?] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.12.jar!/:4.4.12] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.12.jar!/:4.4.12] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.12.jar!/:4.4.12] at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar!/:4.1.4] at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar!/:4.1.4] at java.lang.Thread.run(Thread.java:830) [?:?] [DEBUG] 2020-09-02 17:57:49.351 [pool-7-thread-1] InternalHttpAsyncClient: - [exchange: 801] connection request failed [DEBUG] 2020-09-02 17:57:49.351 [BeaconProcessor] RestClient: - request [POST http://ingestion-es-http:9200/recordmetadata-fei/_update/1599066000000-1599069600000:testapp:ingestiontenant3:firewall.traffic?retry_on_conflict=5&timeout=1m] failed java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:71) ~[httpcore-4.4.12.jar!/:4.4.12] at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:84) ~[httpcore-4.4.12.jar!/:4.4.12] at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70) ~[httpasyncclient-4.1.4.jar!/:4.1.4] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:244) ~[elasticsearch-rest-client-7.7.0.jar!/:7.7.0] at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.7.0.jar!/:7.7.0] at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1609) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0] at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1579) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0] at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1549) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0] at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:1016) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0] at com.paloaltonetworks.cortex.ingestion.beacon.store.es.ElasticSearchStore.update(ElasticSearchStore.java:182) ~[ingest-common-2.0.0.jar!/:?] at com.paloaltonetworks.cortex.ingestion.beacon.store.es.ElasticSearchStore.update(ElasticSearchStore.java:36) ~[ingest-common-2.0.0.jar!/:?] at com.paloaltonetworks.cortex.ingestion.frontend.beacon.ElasticSearchRecordBeaconWriter.write(ElasticSearchRecordBeaconWriter.java:58) ~[classes!/:?] at com.paloaltonetworks.cortex.ingestion.frontend.beacon.ElasticSearchRecordBeaconWriter.write(ElasticSearchRecordBeaconWriter.java:19) ~[classes!/:?] at com.paloaltonetworks.cortex.ingestion.beacon.BeaconWriter.run(BeaconWriter.java:65) ~[ingest-common-2.0.0.jar!/:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?] at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:830) [?:?] Caused by: java.net.ConnectException: Connection refused at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?] at sun.nio.ch.Net.pollConnectNow(Net.java:579) ~[?:?] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820) ~[?:?] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.12.jar!/:4.4.12] at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.12.jar!/:4.4.12] at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.12.jar!/:4.4.12] at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar!/:4.1.4] at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar!/:4.1.4] ... 1 more [DEBUG] 2020-09-02 17:57:49.351 [BeaconProcessor] RestClient: - updated [[host=http://ingestion-es-http:9200]] already in blacklist ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 6, 2020, 10:15pm UTC](https://discuss.elastic.co/t/intermittent-errors-connecting-to-es-cluster-when-using-high-level-rest-client/247929/2 "2020-10-06T22:15:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.