Having issues translating query dsl to NEST

woal_idal · August 16, 2021, 6:37pm

I have a console application that will read the logs from elasticsearch and insert those logs to a database. However I am having an issue with the count api implementation in NEST. Whenever, I type the following in the CLI of kibana I get a value for "count" to be 16.

GET /customer-simulation-es-app-logs-development-2021-08/_count
{
  "query": {
    "bool": {
      "should": [
            {
              "match": {
                "fields.Username": "Jessica12"
              }
            }
      ],
      "filter": [
        {
          "range": {
            "@timestamp": {
                 "gte": "2021-08-16T00:00:00.000-05:00",
                 "lt": "2021-08-16T23:59:59.999-05:00"
            }
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

The results returned are:

{
  "count" : 16,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  }
}

Then in my NEST implementation for .NET, I get countResponse to be 0 instead of 16. I do not understand why I am getting that if I am doing the exact same query search in the CLI. Am I not?

            private IElasticClient _elasticClient = new ElasticClient();

            string indexName = "customer-simulation-es-app-logs-development-2021-08";
            var connectionSettings = new ConnectionSettings(new Uri("http://localhost:9200"));
            connectionSettings.DefaultIndex(indexName);
            connectionSettings.EnableDebugMode();
            _elasticClient = new ElasticClient(connectionSettings);


            // this will tell us how much hits/results there is based on the following criteria 
            var countResponse = _elasticClient.Count<SourceItems>(c => c
                 .Query(q => q
                     .Bool(b => b
                         .Should(
                               m => m
                               .Match(ma => ma
                                   .Field(fa => fa._Source.fields.Username)
                                   .Query("Jessica12")))
                         .Filter(f => f.DateRange(dr => dr
                         .Field("@timestamp")
                             .GreaterThanOrEquals("2021-08-16T00:00:00.000-05:00")
                             .LessThanOrEquals("2021-08-16T23:59:59.999-05:00")))
                         .MinimumShouldMatch(1)))).Count;


            Console.WriteLine($"\nDocuments in index: {countResponse}");

SourceItems.cs:

public class SourceItems
{
    public String _Index { get; set; }
    public String _Id { get; set; }
    public Source _Source { get; set; }
}

public class Source
{
    [Date(Name = "@timestamp")]
    public DateTimeOffset timestamp { get; set; }
    public String level { get; set; }
    public String message { get; set; }
    public EsFields fields { get; set; }
}

public class EsFields
{
    public String Username { get; set; }
    public String IP { get; set; }
    public String SourceContext { get; set; }
    public String RequestId { get; set; }
    public String RequestPath { get; set; }
    public String ConnectionId { get; set; }
    public String CorrelationId { get; set; }
}

When I do a search to match the username Jessica12 I get the hits and you can see the username stored in the field.

GET /customer-simulation-es-app-logs-development-2021-08/_search
{
  "size": 200,
  "query": {
    "bool": {
      "should": [
            {
              "match": {
                "fields.Username": "Jessica12"
              }
            }
      ],
      "filter": [
        {
          "range": {
            "@timestamp": {
                 "gte": "2021-08-16T00:00:00.000-05:00",
                 "lt": "2021-08-16T23:59:59.999-05:00"
            }
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

stevejgordon · August 17, 2021, 1:19pm

Hi @woal_idal. Your query looks generally okay, but your match definition is accessing "_Source.fields.Username", not "fields.Username" as shown in your Kibana query.

To match 1-to-1 it looks like perhaps this is what you want?

var countResponse = client.Count<Source>(c => c
	.Query(q => q
		.Bool(b => b
			.Should(
				m => m
				.Match(ma => ma
					.Field(fa => fa.fields.Username)
					.Query("Jessica12")))
			.Filter(f => f.DateRange(dr => dr
				.Field("@timestamp")
					.GreaterThanOrEquals("2021-08-16T00:00:00.000-05:00")
					.LessThanOrEquals("2021-08-16T23:59:59.999-05:00")))
			.MinimumShouldMatch(1)))).Count;

You don't need to model types for the top-level JSON in the Elasticsearch hits. Your C# model should relate the raw document which is represented within the _source field in the JSON. So for example, your model might look like this:

public class LogEntry
{
	[Date(Name = "@timestamp")]
	public DateTimeOffset Timestamp { get; set; }
	public string Level { get; set; }
	public string Message { get; set; }
	public Fields Fields { get; set; }
}

public class Fields
{
	public string Username { get; set; }
	public string IP { get; set; }
	public string SourceContext { get; set; }
	public string RequestId { get; set; }
	public string RequestPath { get; set; }
	public string ConnectionId { get; set; }
	public string CorrelationId { get; set; }
}

You could then switch the generic type for the Count request to to get type inference.

woal_idal · August 17, 2021, 2:09pm

stevejgordon:

var countResponse = client.Count<Source>(c => c
	.Query(q => q
		.Bool(b => b
			.Should(
				m => m
				.Match(ma => ma
					.Field(fa => fa.fields.Username)
					.Query("Jessica12")))
			.Filter(f => f.DateRange(dr => dr
				.Field("@timestamp")
					.GreaterThanOrEquals("2021-08-16T00:00:00.000-05:00")
					.LessThanOrEquals("2021-08-16T23:59:59.999-05:00")))
			.MinimumShouldMatch(1)))).Count;

Hi @stevejgordon. Thank you for the reply, that would make sense! I implemented as you mentioned but for some reason I still am getting count as 0. I made sure that the indexName was correct but it still returns 0. Do you know what is causing this?

This is what I have currently in my console app:

InsertLogData.cs:

               string indexName = "customer-simulation-es-app-logs-development-2021-08";
                var connectionSettings = new ConnectionSettings(new Uri("http://localhost:9200"));
                connectionSettings.DefaultIndex(indexName);
                connectionSettings.EnableDebugMode();
                _elasticClient = new ElasticClient(connectionSettings);


                // this will tell us how much hits/results there is based on the following criteria 
                var countResponse = _elasticClient.Count<LogEntry>(c => c
                        .Query(q => q
                            .Bool(b => b
                                .Should(
                                    m => m
                                    .Match(ma => ma
                                        .Field(fa => fa.Fields.Username)
                                        .Query("Jessica12")))
                                .Filter(f => f.DateRange(dr => dr
                                    .Field("@timestamp")
                                        .GreaterThanOrEquals("2021-08-16T00:00:00.000-05:00")
                                        .LessThanOrEquals("2021-08-16T23:59:59.999-05:00")))
                                .MinimumShouldMatch(1)))).Count;


                Console.WriteLine($"\nDocuments in index: {countResponse}");

Update, I tried changing the <LogEntry> to <EsSource> and remaining the first class to EsSource instead of LogEntry and I still get the same issue. I believe my main issue is here:

public class EsSource
    {
        [Date(Name = "@timestamp")]
        public DateTimeOffset timestamp { get; set; }
        public String level { get; set; }
        public String message { get; set; }
        public Fields fields { get; set; }
    }

    public class Fields
    {
        public String Username { get; set; }
        public String IP { get; set; }
        public String SourceContext { get; set; }
        public String RequestId { get; set; }
        public String RequestPath { get; set; }
        public String ConnectionId { get; set; }
        public String CorrelationId { get; set; }

    }

var countResponse = _elasticClient.Count<EsSource>(c => c
                     .Query(q => q
                         .Bool(b => b
                             .Should(
                                   m => m
                                   .Match(ma => ma
                                       .Field(fa => fa.fields.Username)
                                       .Query("Jessica12")))
                             .Filter(f => f.DateRange(dr => dr
                             .Field("@timestamp")
                                 .GreaterThanOrEquals("2021-08-16T00:00:00.000-05:00")
                                 .LessThanOrEquals("2021-08-16T23:59:59.999-05:00")))
                             .MinimumShouldMatch(1)))).Count;

It returns 0. But if I do not query username and jessica12 and I query lets say Ievel, Information I get the count value that I get in the CLI. How come I cannot query the username field with jessica12 if I can in the CLI?

in the CLI with jessica:

stevejgordon · August 17, 2021, 2:52pm

I've just spotted that within Fields they are defined using PascalCase, so it's "Username". The NEST field name inferred defaults to camelCase, so it's translating your request so the should clause looks like this:

":[{"match":{"fields.username":{"query":"Jessica12"}}}]

There are a few solutions to this but the simplest, for now, is to add attributes to the properties with the correct name, as you did for Timestamp. e.g.

[Text(Name = "Username")]
public string Username { get; set; }

You can read more about inference in our docs.

woal_idal · August 17, 2021, 2:55pm

You are the best!! I appreciate it, changing that in the model class like I did for timestamp worked! How would you know when you should add the attributes like [Text(Name = "Username")] to the properties?

stevejgordon · August 18, 2021, 9:05am

You're welcome. Those docs I linked to provide more detail, but in short, NEST infers the JSON name for fields and by default presumes they will be camel-cased as its the convention. As such, when creating the request and using field inference, the first letter is lowercase. If that behaviour is not valid for some types, you can use attributes to change what the JSON name will be. For more complex scenarios you can change how inference works for all fields. In your case, as the data has a mix of casing, it probably makes sense to attribute those properties which need to specifically be pascal cased.

system · September 15, 2021, 9:05am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.