Connection Reset By Peer IOException

Elastic Stack Elasticsearch
1

Hi All,

We are getting Connection reset by peer issue with ElasticSearch Rest Client connection.
Below issue is faced with both single & multi node cluster setup.
Can someone please help in resolving this issue?

Thanks in advance.

ES version - 7.4
ES Client used - org.elasticsearch.client.RestHighLevelClient

Exception Logs:

2020-09-03T18:37:45.056-07:00 java.io.IOException: Connection reset by peer

2020-09-03T18:37:45.056-07:00 at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:793)

2020-09-03T18:37:45.056-07:00 at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)

2020-09-03T18:37:45.057-07:00 at org.elasticsearch.client.RestClient.performRequest(RestClient.java:221)

2020-09-03T18:37:45.057-07:00 at org.elasticsearch.client.RestClient.performRequest(RestClient.java:221)

2020-09-03T18:37:45.057-07:00 at org.elasticsearch.client.RestClient.performRequest(RestClient.java:205)

2020-09-03T18:37:45.057-07:00 at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1454)

2020-09-03T18:37:45.057-07:00 at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1424)

2020-09-03T18:37:45.057-07:00 at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1394)

2020-09-03T18:37:45.057-07:00 at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:930)

2020-09-03T18:37:45.059-07:00 at com.zibo.notification.dao.BaseEntityDaoImpl.search(BaseEntityDaoImpl.java:174)

2020-09-03T18:37:45.059-07:00 at com.zibo.notification.dao.BaseEntityDaoImpl.findByAttribute(BaseEntityDaoImpl.java:164)

2020-09-03T18:37:45.059-07:00 at com.zibo.notification.dao.NotificationDaoImpl.getNotificationByMessageId(NotificationDaoImpl.java:51)

2020-09-03T18:37:45.059-07:00 at com.zibo.notification.service.impl.NotificationServiceImpl.getByMessageId(NotificationServiceImpl.java:52)

2020-09-03T18:37:45.059-07:00 at com.zibo.notification.lambda.validator.MessageValidatorImpl.validateForDuplicateMessage(MessageValidatorImpl.java:99)

2020-09-03T18:37:45.059-07:00 at com.zibo.notification.lambda.validator.MessageValidatorImpl.validateSQSMessage(MessageValidatorImpl.java:61)

2020-09-03T18:37:45.059-07:00 at com.zibo.notification.service.impl.SQSMessageProcessorImpl.handleSQSEvent(SQSMessageProcessorImpl.java:62)

2020-09-03T18:37:45.059-07:00 at com.zibo.notification.lambda.handler.SQSEventHandler.handleRequest(SQSEventHandler.java:40)

2020-09-03T18:37:45.059-07:00 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

2020-09-03T18:37:45.059-07:00 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

2020-09-03T18:37:45.059-07:00 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

2020-09-03T18:37:45.059-07:00 at java.lang.reflect.Method.invoke(Method.java:498)

2020-09-03T18:37:45.059-07:00 at lambdainternal.EventHandlerLoader$PojoMethodRequestHandler.handleRequest(EventHandlerLoader.java:263)

2020-09-03T18:37:45.059-07:00 at lambdainternal.EventHandlerLoader$PojoHandlerAsStreamHandler.handleRequest(EventHandlerLoader.java:180)

2020-09-03T18:37:45.059-07:00 at lambdainternal.EventHandlerLoader$2.call(EventHandlerLoader.java:902)

2020-09-03T18:37:45.059-07:00 at lambdainternal.AWSLambda.startRuntime(AWSLambda.java:341)

2020-09-03T18:37:45.059-07:00 at lambdainternal.AWSLambda.(AWSLambda.java:63)

2020-09-03T18:37:45.059-07:00 at java.lang.Class.forName0(Native Method)

2020-09-03T18:37:45.059-07:00 at java.lang.Class.forName(Class.java:348)

2020-09-03T18:37:45.059-07:00 at lambdainternal.LambdaRTEntry.main(LambdaRTEntry.java:150)

2020-09-03T18:37:45.059-07:00 Suppressed: java.io.IOException: Connection reset by peer

2020-09-03T18:37:45.059-07:00 at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:793)

2020-09-03T18:37:45.059-07:00 at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)

2020-09-03T18:37:45.059-07:00 ... 26 more

2020-09-03T18:37:45.059-07:00 Suppressed: java.io.IOException: Connection reset by peer

2020-09-03T18:37:45.059-07:00 at org.elasticsearch.client.RestClient.extractAndWrapCause(RestClient.java:793)

2020-09-03T18:37:45.059-07:00 at org.elasticsearch.client.RestClient.performRequest(RestClient.java:218)

2020-09-03T18:37:45.059-07:00 ... 25 more

2

This means that something other than the client closed the connection. However it wasn't Elasticsearch either: Elasticsearch keeps client connections open until the client closes them. This means whatever closed the connection, it must be between the client and Elasticsearch. Unfortunately there's not a lot more we can say in terms of Elasticsearch, you'll need to seek some advice from network engineering folks if you need further help here.

3

Thanks David.
I have some doubts on the ES coding best practices that we need to follow to ensure that connection reset issue is not caused by poor programming.

The custom implementation that we have done for ES connection using HighLevelRestClient is not explicitly closing the connection programmatically.

Found below class level comments in HighLevelRestClient:
"* High level REST client that wraps an instance of the low level {@link RestClient} and allows to build requests and read responses. The

  • {@link RestClient} instance is internally built based on the provided {@link RestClientBuilder} and it gets closed automatically when
  • closing the {@link RestHighLevelClient} instance that wraps it."

Does it mean that we have to call close() method on HighLevelRestClient instance to close the connection within our codebase?
Or whether the HighLevelRestClient will automatically close the connection after some defined time-duration?

Thanks,
Surabhi

4

Opening a client is expensive (i.e. takes quite a bit of time and effort) so most applications typically do it only once, at startup. It therefore follows that most applications typically only close the client once, at shutdown.

No, as I said, the close of this connection was not the client's doing. The connection was reset by a peer, i.e. something else. The client itself does not have a expiry time after which it must be closed.

5

Although this is certainly a network issue, it is possible that having your client send TCP keepalives would work around it. Today this doesn't happen by default, but you can enable them. See this issue for more details:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.