Elastic Search - Error Detail - Only one usage of each socket address (protocol/network address/port) is normally permitted Error

[ankurpatel (ankur)]

We have got below error in API which access data from Elastic Search. Error generated when web site traffic is high. We have configured Elastic Search on Single Node. We developed API(.NET core) by using NEST (7.15.1 library)

Please help us as its a production issue.
Our production serer configuration is:

• Operating System-Ubuntu 18.04 LTS
• Processor - Intel Xeon Gold 6140 – 12Core
• RAM-128GB
• HDD-500GB SSD

Error Detail:

FailureReason: FailedProductCheck while attempting POST on http://XX.XXX.XXX.XXX:XXXX/searchkeyword/_search?pretty=true&typed_keys=true

Audit trail of this API call:

OriginalException: System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted.

---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted. at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)

Request:

<Request stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on ConnectionSettings to force it to be set on the response.>

Response:

Inner Exception: The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call. Some functionality may not be compatible if the server is running an unsupported product.

Elasticsearch.Net.PipelineException: The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call. Some functionality may not be compatible if the server is running an unsupported product. ---> System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted. ---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted. at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData) --- End of inner exception stack trace --- at Elasticsearch.Net.RequestPipeline.ThrowIfTransientProductCheckFailure() at Elasticsearch.Net.RequestPipeline.Ping(Node node) at Elasticsearch.Net.Transport1.Ping(IRequestPipeline pipeline, Node node) at Elasticsearch.Net.Transport`1.Request[TResponse](HttpMethod method, String path, PostData data, IRequestParameters requestParameters)

Exception:

Elasticsearch.Net.ElasticsearchClientException: The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call. Some functionality may not be compatible if the server is running an unsupported product. Call: Status code unknown from: GET /?pretty=true ---> Elasticsearch.Net.PipelineException: The client is unable to verify that the server is Elasticsearch due to an unsuccessful product check call. Some functionality may not be compatible if the server is running an unsupported product. ---> System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted. ---> System.Net.Sockets.SocketException (10048): Only one usage of each socket address (protocol/network address/port) is normally permitted. at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts) at Elasticsearch.Net.HttpConnection.Request[TResponse](RequestData requestData)

We tried following thing to fix this issue but not succeed.

1. We have reset connection time out - 30 second and connection limit to 500.
2. Changed Elasticsearch api port no 9800 to 9801.
3. We haver changed public IP address to private IP address.

[stevejgordon (Steve Gordon)]

Hi, @ankurpatel.

This exception is being thrown from the .NET networking stack and is not a client issue. I've personally never hit this, but it's typically thrown if your device runs out of ephemeral ports and is, therefore, unable to select a port for the new connection.

Are you reusing the same ElasticClient instance across your application? If you're using DI, it should be registered as a singleton lifetime. By reusing the same instance, the underlying HTTP connections are reused, which is essential. Otherwise, many instances can lead to short-lived connections and ports being tied up.

If you are using HttpClient elsewhere in your application, the same rule applies, and you should reuse the same instance to avoid socket exhaustion.

I have a couple of blog posts on this subject which could help:

• HttpClient Connection Pooling in .NET Core - Steve Gordon - Code with Steve
• HttpClientFactory in ASP.NET Core 2.1 (Part 1) - Steve Gordon - Code with Steve

[ankurpatel (ankur)]

Thank you very much Steve. We have changed implementation logic as per your suggestion and fixed the issue.

Big Thanks.......