NEST Client - how to resolve elastic search client exceptions please

krishnapss · September 24, 2019, 7:20pm

HI,
Could someone please help me to resolve this issue.
We are recently getting Two types of exceptions
Elasticsearch.Net.UnexpectedElasticsearchClientException
The operation was canceled.
Elasticsearch.Net.ElasticsearchClientException
Maximum timeout reached while retrying request. Call: Status code unknown from: POST /indexname/_search?typed_keys=true
We are using Elastic cloud service and we have created asp.net core 2.2 API to communicate with Elasticsearch. For communication we are using NEST Client 7.0. Our service is working ok for some time and started throwing exceptions until we restart the our api server. But if we try to connect to Elastic cloud service from post man directly it is working ok.
Exception details

Elasticsearch.Net.UnexpectedElasticsearchClientException:
at Elasticsearch.Net.Transport1.Request (Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d)
Inner exception System.OperationCanceledException handled at Elasticsearch.Net.Transport1.Request:
at System.Net.Http.HttpClient.HandleFinishSendAsyncError (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Net.Http.HttpClient+d__63.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at Elasticsearch.Net.HttpConnection.Request (Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d)
at Elasticsearch.Net.RequestPipeline.CallElasticsearch (Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d)
at Elasticsearch.Net.Transport1.Request (Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d)

Elasticsearch.Net.ElasticsearchClientException:
at Elasticsearch.Net.Transport1.HandleElasticsearchClientException (Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d)
at Elasticsearch.Net.Transport1.FinalizeResponse (Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d)
at Elasticsearch.Net.Transport1.Request (Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d)

Inner exception System.Threading.Tasks.TaskCanceledException handled at Elasticsearch.Net.Transport1.HandleElasticsearchClientException:
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Threading.Tasks.ValueTask1.get_Result (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpConnectionPool+d__39.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.RedirectHandler+d__4.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable1+ConfiguredTaskAwaiter.GetResult (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Net.Http.HttpClient+d__63.MoveNext (System.Net.Http, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification
at Elasticsearch.Net.HttpConnection.Request (Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d)

I am using singleton ElasticClient in my code. I have created a factory class which is a registered as singleton.
Elastic client creation code

internal class ElasticClientFactory 
    {
        private readonly IOptions<Configuration> _configuration;
        private ElasticClient _client;
        public ElasticClientFactory(IOptions<Configuration> elasticSearchConfiguration)
        {                
            _client = CreateClient();
        }
        private ElasticClient CreateClient()
        {
            var settings = new ConnectionSettings(new Uri(_configuration.Value.Url))
                .DefaultMappingFor<Model>(m => m.IdProperty(p => p.Id))
                .DefaultFieldNameInferrer(f => f.ToLower())
                .ThrowExceptions();
            settings.BasicAuthentication(_configuration.Value.Username, _configuration.Value.Password);

            return new ElasticClient(settings);
        }
        public IElasticClient Create()
        {
            return _client ?? (_client = CreateClient());
        }
    }

forloop · September 26, 2019, 7:14am

Based on

It looks like the request is timing out from the client i.e. the request is taking longer to complete than the client has been configured to wait for, so it cancels the request.

By default, the request timeout is set to 1 minute, but this can be changed on ConnectionSettings

var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var settings = new ConnectionSettings(pool)
    .RequestTimeout(TimeSpan.FromMinutes(5));

var client = new ElasticClient(settings);

or on a per request basis

var searchResponse = client.Search<object>(s => s
	.RequestConfiguration(r => r
		.RequestTimeout(TimeSpan.FromMinutes(5))
	)
);

If you're running against Elastic Cloud, you may also want to use the CloudConnectionPool, which configures the client for communicating with Elastic Cloud.

krishnapss · September 26, 2019, 7:33pm

Hi Russ,
Thank you very much for your help. I will increase the timeout and use the CloudConnectionPool.
But is increasing the timeout best solution please? (the user response time)?
What are the known reasons that can cause this timeout please? e.g. do you think may be the number of requests? And also what is the best way to identify and set up alerts for this type of problems.

Thank you,
Krishna

forloop · September 27, 2019, 5:02am

There's many layers to this question to unpack! Some requests may naturally take longer than others, depending on complexity and type of operation. Some of these requests may take longer than the default request timeout of the client of 1 minute so for these you may want to set a longer timeout.

The client sets a default timeout so that it doesn't potentially wait forever, keeping a connection open and blocking other outbound connections from being initiated.

If you haven't done so already, it's worth configuring monitoring for your cluster so you can observe the health of the cluster over time.

krishnapss · September 27, 2019, 9:30am

Thank you Russ.

system · October 25, 2019, 9:30am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.