Elasticsearch version ( bin/elasticsearch --version ):
7.6.2
Plugins installed :
JVM version ( java -version ):
sh-4.2# /usr/share/elasticsearch/jdk/bin/java -version
openjdk version "13.0.2" 2020-01-14
OpenJDK Runtime Environment AdoptOpenJDK (build 13.0.2+8)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 13.0.2+8, mixed mode, sharing)
OS version ( uname -a if on a Unix-like system):
Linux x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior :
Steps to reproduce :
I am running a 3 node ES cluster on Kubernetes and performing about 80 upserts per minute. When connecting via the High Level Rest client, I see intermittent errors (see below). Other times, the updates work fine.
When I look at my CPU usage, the hot cpu is running at 15% usage.
I am not sure if the errors are with the cluster or with the Rest client.
Rest Client initialization :
private RestHighLevelClient initClient(ElasticSearchStoreConfig config) throws IOException {
String password = "foobar";
final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
credentialsProvider.setCredentials(AuthScope.ANY,
new UsernamePasswordCredentials(config.getUserName(), password));
RestClientBuilder builder = RestClient.builder(new HttpHost(config.getHostname(),
config.getPort(), config.getElasticSearchScheme()))
.setHttpClientConfigCallback(httpClientBuilder ->
httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));
return new RestHighLevelClient(builder);
}
I am running the update as follows (there is a retry policy):
UpdateRequest updateRequest = mapper.update(key, data);
UpdateResponse response = esClient.update(updateRequest, RequestOptions.DEFAULT);
Provide logs (if relevant) :
[DEBUG] 2020-09-02 17:57:49.351 [pool-7-thread-1] PoolingNHttpClientConnectionManager: - Connection request failed
java.net.ConnectException: Connection refused
at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
at sun.nio.ch.Net.pollConnectNow(Net.java:579) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820) ~[?:?]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at java.lang.Thread.run(Thread.java:830) [?:?]
[DEBUG] 2020-09-02 17:57:49.351 [pool-7-thread-1] InternalHttpAsyncClient: - [exchange: 801] connection request failed
[DEBUG] 2020-09-02 17:57:49.351 [BeaconProcessor] RestClient: - request [POST http://ingestion-es-http:9200/recordmetadata-fei/_update/1599066000000-1599069600000:testapp:ingestiontenant3:firewall.traffic?retry_on_conflict=5&timeout=1m] failed
java.util.concurrent.ExecutionException: java.net.ConnectException: Connection refused
at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:71) ~[httpcore-4.4.12.jar!/:4.4.12]
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:84) ~[httpcore-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:244) ~[elasticsearch-rest-client-7.7.0.jar!/:7.7.0]
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:235) ~[elasticsearch-rest-client-7.7.0.jar!/:7.7.0]
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1609) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0]
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1579) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0]
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1549) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0]
at org.elasticsearch.client.RestHighLevelClient.update(RestHighLevelClient.java:1016) ~[elasticsearch-rest-high-level-client-7.7.0.jar!/:7.7.0]
at com.paloaltonetworks.cortex.ingestion.beacon.store.es.ElasticSearchStore.update(ElasticSearchStore.java:182) ~[ingest-common-2.0.0.jar!/:?]
at com.paloaltonetworks.cortex.ingestion.beacon.store.es.ElasticSearchStore.update(ElasticSearchStore.java:36) ~[ingest-common-2.0.0.jar!/:?]
at com.paloaltonetworks.cortex.ingestion.frontend.beacon.ElasticSearchRecordBeaconWriter.write(ElasticSearchRecordBeaconWriter.java:58) ~[classes!/:?]
at com.paloaltonetworks.cortex.ingestion.frontend.beacon.ElasticSearchRecordBeaconWriter.write(ElasticSearchRecordBeaconWriter.java:19) ~[classes!/:?]
at com.paloaltonetworks.cortex.ingestion.beacon.BeaconWriter.run(BeaconWriter.java:65) ~[ingest-common-2.0.0.jar!/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) ~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
at sun.nio.ch.Net.pollConnectNow(Net.java:579) ~[?:?]
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:820) ~[?:?]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvent(DefaultConnectingIOReactor.java:174) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.DefaultConnectingIOReactor.processEvents(DefaultConnectingIOReactor.java:148) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:351) ~[httpcore-nio-4.4.12.jar!/:4.4.12]
at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:221) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64) ~[httpasyncclient-4.1.4.jar!/:4.1.4]
... 1 more
[DEBUG] 2020-09-02 17:57:49.351 [BeaconProcessor] RestClient: - updated [[host=http://ingestion-es-http:9200]] already in blacklist