NEST is much slower than Kibana

[hhkx (Hasan)]

Hi people,

We have been testing Elastic Search for our logging system. There is 300 thousand data on an index in ElasticSearch cluster.

I have installed both Kibana and NEST on my PC. The cluster on a local server.

When I make a query from Kibana using Dev Tools, the response time is 1 ms and it brings 10 records

GET /dbmessages/_search
{
  "query": {
"bool": {
  "must": [
    { "match": { "user": "Robin*" } }
    
  ]
}
  }
}

However with .NET client (using NEST), the response time is around 600 ms, even with the same query.

var sw = new Stopwatch();
sw.Start();    

var result = client.Search<Message>((s => s
        .Index("dbmessages")
        .From(0)
        .Size(10)
        .Query(q => q.Raw(@"  ""query"": { ""bool"": { ""must"": [ { ""match"": { ""user"": ""Robin*"" } } ] } }"))));

sw.Stop();
Console.WriteLine(sw.ElapsedMilliseconds);

Even mongoDb and SQLServer make select query faster. I believe something is wrong.

Anyone has any idea why this happens?

Thanks

[forloop (Russ Cam)]

@hhkx, I gave some guidance on this in a comment on the same question you asked on stack overflow, I'll add here for posterity.

Bear in mind that the first query to Elasticsearch from NEST will be much slower because the client caches a lot of delegates, compiled expressions and json serialization properties on this first call.

When running these kinds of benchmarks, it is advisable to run in release mode, prime caches first (make calls and discard results), send many calls and take the percentiles of such calls. A tool such as BenchmarkDotNet can help here.

Additionally, with the NEST call, you'll also be including serialization of request and deserialization of response.

With all of that said, I would expect NEST to be relatively slower overall in making a request compared to using Console, because the client needs to

1. serialize a strongly typed request into the byte representation of the request json
2. make the request to Elasticsearch
3. deserialize the response bytes into a strongly typed response

Our benchmarking and profiling in the profiling branch indicates that a relatively large amount of time is spent in serialization/deserialization. We have intentions to generate custom serializers for each request/response to improve this in the future.

Whilst I would expect NEST to be slower, it should be in the same order of magnitude (taking into account the information above). Performance is a feature we care about but there are many other features that make using the client a good choice. Here's a few

1. Strong types for all requests and responses provides compile time type safety though the use of member access lambda expressions
2. Cluster sniffing and node failover to automatically retry requests against another node in the cluster when it fails for a retry-able reason
3. Inference for field mappings, ids, indices and types from C# POCOs

The NEST client also exposes the low level Elasticsearch.Net client through the .LowLevel property. Using the low level client, you can send json strings, anonymous types or bytes for the request and specify the response type too. Here's an example

var client = new ElasticClient();

    // the generic type parameter specifies the response type. Here, we use string
ElasticsearchResponse<string> response = client.LowLevel.Search<string>(
    "index",
    "type",
    new
    {
      query = new
      {
    	  match_phrase = new
    	  {
    		  body = "this should never happen"
    	  }
      }
    });

// this will be a json string
var body = response.Body;

[hhkx (Hasan)]

Hi @forloop,

Thanks for your reply. It is very understandable.

Well, is there a way to find out process time on the elasticsearch side within the NEST result?

.//edit: Okay, I tried the code you wrote above and it showed me the exactly what I want. Thank you!