Elasticsearch search query with filter terms having more than 11 field names not returning proper result for 12th fieldname

Elastic Stack Elasticsearch
1

Elasticsearch search query with filter terms having more than 11 field names not returning proper result for 12th fieldname

{
          "size":0,
          "aggs":{
             "filtered_data":{
                "filter":{
                   "bool":{
                      "must":[
                         {
                            "term":{
                               "org_code.keyword": "acme"
                            }
                         },
                         {
                            "terms":{
                               "dp_name.keyword":[
                                  "fieldname1",
                                  "fieldname2",
                                  "fieldname3",
                                  "fieldname4",
                                  "fieldname5",
                                  "fieldname6",
                                  "fieldname7",
                                  "fieldname8",
                                  "fieldname9",
                                  "fieldname10",
                                  "fieldname11",
                                  "fieldname12"
                               ]
                            }
                         },
                         {
                            "range":{
                               "@timestamp":{
                                  "gte":"2021-01-01T00:00:00.000Z",
                                  "lte":"2023-08-09T09:40:00.000Z"
                               }
                            }
                         }
                      ]
                   }
                },
                "aggs":{
                   "group_by_timestamp":{
                      "date_histogram":{
                         "field":"@timestamp",
                         "calendar_interval":"1y"
                      },
                      "aggs":{
                         "group_by_dc_code":{
                            "terms":{
                               "field":"dc_code.keyword"
                            },
                            "aggs":{
                               "group_by_dp_name":{
                                  "terms":{
                                     "field":"dp_name.keyword"
                                  },
                                  "aggs":{
                                     "value_stats":{
                                        "stats":{
                                           "field":"value"
                                        }
                                     },
                                     "unit":{
                                        "terms":{
                                           "field":"unit.keyword"
                                        }
                                     }
                                  }
                               }
                            }
                         }
                      }
                   }
                }
             }
          }
        }

The result of 12th field 'fieldname12' is not coming correctly, the elasticsearch index has 7 records for this field but returning only 2 of them.

Please suggest how to make the above query work properly.

Using Elasticsearch version 8.7 for the above.

2

Hi @ramanm
Try the query without the date range... see if you get the same result...there is nothing magic about 12 terms as far as I now...

                        {
                            "range":{
                               "@timestamp":{
                                  "gte":"2021-01-01T00:00:00.000Z",
                                  "lte":"2023-08-09T09:40:00.000Z"
                               }
                            }
                         }
3

Thanks for the reply @stephenb. I had removed the date range but the query still giving just the 2 result for one of the attribute which becomes the 12th attribute upon sorting the field name list. If we remove any one of the field (e.g,. fieldname5) from the list then the query works fine with 12th fieldname is fetching the all its values as well. Please let me know what else could be the issue.

4

It seems like you are running a nested aggregation without specifying the size anywhere. By default only 10 terms are returned for term aggregations. As you have 12 values some may be missing from the results. Does the behaviour change if you e.g. specify a size of 12 for the dp_name aggregation and another suitable value for dc_code (depending on the cardinality of this field)?

1 Like
5

Thanks @Christian_Dahlqvist . You are correct. I updated the size with higher value it started working in my postman client. I used the same query using my nodejs elastic client library but giving same problem as originally stated. The result of 12th field 'fieldname12' is not coming correctly, the Elasticsearch index has 7 records for this field but returning only 2 of them. Please provide inputs to overcome this issue. Thanks.

6

If it works in Postman it seems to indicate that it is the size not being set that is indeed the issue. I suspect you are doing something wrong when using the node.js client, but am not a javascript developer.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.