NEST Client opens new connection for any request

Elastic Stack Elasticsearch
1

Hello there and welcome Elastic community,

we are using Elastic search with NEST 5.5. One thing we realized is, that the Elastic Client seems to open a new TCP connection for every request. One of our usecases uses the Elastic Search Scroll mechanism in which all results are received via scrolling AND multisearch.

In my stress tests, we request around 55.000 results in packages of 1000. All requests are done sequentially so the order is somethine like Scroll -> Wait for Results -> Multisearch -> Wait for Results -> Scroll -> Wait for Results... etcpp. Overall for a single test we have 55 scrolls and 55 msearches. This is all done via the high level client (NEST.ElasticClient).
This test is repeated 5 times, which results in 550 total requests. When I observe our TCP connections via "TCPView" I can see a lot of connections opening to our server, which all end up in state "Time_Wait". Typically after our tests we have around 400 - 450 connections in this state.

After googling a lot and even reading some Threads in this forum, TCP Keep Alive seems to be the relevant point. However, after changing a lot of settings, the observed behaviour does not change.

So, first of all I am going to present the relevant code, which is creating our client:

var connectionPool = new SingleNodeConnectionPool(new Uri(esUri));
  var connection = new EsHttpConnection(mCurrentAccount, mAccountAccess);
  mEsClientConnectionSettings = new ConnectionSettings(connectionPool, connection, new SerializerFactory((settings, values) =>
  {
    settings.NullValueHandling = NullValueHandling.Ignore;
    settings.TypeNameHandling = TypeNameHandling.None;
  }));

  //If Proxy detection is not disabled, we sometimes have a severe performance issue, when creating httpconnections
  mEsClientConnectionSettings.DisableAutomaticProxyDetection();
  mEsClientConnectionSettings.EnableTcpKeepAlive(TimeSpan.FromMilliseconds(2000), TimeSpan.FromMilliseconds(2000));
  mEsClientConnectionSettings.EnableHttpCompression();
  mEsClientConnectionSettings.RequestTimeout(TimeSpan.FromMilliseconds(mAccountAccess.RequestTimeout));
  
  mEsClient = new ElasticClient(mEsClientConnectionSettings);

  Debug.WriteLine("Created new client");
  return mEsClient;

Our client is only created once and then reused. I verified this behaviour with the help of the Debug.WriteLine() line at the end.

The used EsHttpConnection is a derived class of HttpConnection. It has some bool properties (which are substituted in the snippet) and overrides the CreateHttpWebRequest:

protected override HttpWebRequest CreateHttpWebRequest(RequestData requestData)
{
  Stopwatch sw = Stopwatch.StartNew();
  HttpWebRequest request = base.CreateHttpWebRequest(requestData);
  var elapsed = sw.ElapsedMilliseconds;
  if (elapsed > 1000)
  {
    Log.Warning(String.Format(@"Creating HttpWebRequest for Elastic search took more than 1000ms. Measured time: {0}ms", elapsed));
  }
// This happens especially, if we dont use mEsClientConnectionSettings.DisableAutomaticProxyDetection();
 
  ...

  request.PreAuthenticate = true;
  request.AllowAutoRedirect = true;
  request.UseDefaultCredentials = false;

...

  request.UseDefaultCredentials = true;
  request.Credentials = CredentialCache.DefaultCredentials;

  if (true)
  {
    string requestId = Guid.NewGuid().ToString();
    Log.Verbose("Send request {@method} {@uri} with request ID {@requestId}",
      request.Method, request.RequestUri.AbsoluteUri, requestId);

    request.Headers.Add(UriHelper.RequestIdHeaderName, requestId);
  }

  if (true)
  {
    DateTime now = DateTime.Now;
    string timeStamp = now.ToString("hh:mm:ss.fff");
    Log.Verbose("Send request {@method} {@uri} with timestamp {@timeStamp}",
      request.Method, request.RequestUri.AbsoluteUri, timeStamp);

    request.Headers.Add(UriHelper.RequestTimestampHeaderName, timeStamp);
  }
  return request;
}

Does any of you experts see something wrong in creating the request and client?

  • I tried the static ServicePointManager to set TCPKeepAlive
  • our RequestData object has data for keep alive (which is set in the HTTPConnection AlterServicePoint method)
  • The first http request is sending the keep-alive flag in its header, but not the following. This seems to be normal as we use HTTP Protocol Version 11.
  • TCP Timeout on IIS is 120 seconds

Our server is running on IIS8.5 with Windows Server 2012R2.

First Edit:
Using .Net 4.6.2

Typical line in TCPView which are several hundred with raising port number:
[System Process] 0 TCP OurSourceAddress 51358 OurDestinationAddress 80 TIME_WAIT

Thanks in advance,

Jörg

2

NEST will reuse connections. The default connection limit on Desktop CLR is 80, but this can be changed if needed on ConnectionSettings.

There are integration tests to assert that connections are reused, as we have seen connection bleeding happening on .NET Core on Linux with the System.Net.Http.CurlHandler implementation.

It might be worth enabling Network tracing too, to see if it correlates with TCPView.

Some general questions:

  • Have you tried other values for EnableTcpKeepAlive?
  • Is it possible that the connections remain open as the response streams are not being disposed?
3

Hello Russ,

thanks for the reply and sorry for my late response.

With the help of the Nodestats and the internal .Net network tracing, I was able to figure the component causing my problem.

Our Elastic Client connects to an intermediate server to check several access rights. This server then sends the request to the Elastic Server.
If I am connecting directly to the Elastic Server, the connection issue does not occur. So the error lies within our intermediate server implementation and the Elastic Client seems to do fine!

Therefore this is no longer an Elastic Client issue, but one caused by our own implementation.

Kind regards

4

Thanks for reporting back @Sauter_Vector. Glad you were able to get to the bottom of it :+1:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.