Why don't the queries built by Kibana in discover use must?

[Vincent92 (vincent)]

Hello,

I am developing a web application to visualize data in an elastic and I am inspired by the queries made by Kibana in the Discover application by going to inspect the query object created in the Inspect tab of Discover.

I get the impression that apart from the negative terms that use "bool":{"must_not":[]} to create the query, the must[] query (which is not the same thing as must_not[] anyway) is never used and instead Kibana uses a nesting of filter[] query and should[] query.

Is there any particular reason not to use the must[] query, especially if the query contains only "AND " operators?

Moreover, queries containing only "AND " operators have exactly the same nesting structure of should[] as queries containing only "OR " operators. So how does Kibana separate "I want this and this and this " from "I want this or this or this ".

In my previous query interpreter I had a should[] object that contained several must[] objects. Each must[] object stored the "AND " operators side by side, so each time there was an "OR " operator I created a new must[] object in the main should[] object.

Now that I want to use the same structure as the queries made by Kibana's Discover application, I see that the parent object is a filter[] followed by a nesting of should[], so how can I separate an "AND " from an "OR "?

Example of a query created by Kibana retrieved from the Inspect tab of the Discover application for the following query:
field1:"match_field1" AND field2:"match_field2" AND field3:"match_field3" AND field4:"match_field4" AND field5:"match_field5"
This gives the query:

"query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "match_phrase": {
                        "field1": "match_field1"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              },
              {
                "bool": {
                  "filter": [
                    {
                      "bool": {
                        "should": [
                          {
                            "match_phrase": {
                              "field2": "match_field2"
                            }
                          }
                        ],
                        "minimum_should_match": 1
                      }
                    },
                    {
                      "bool": {
                        "filter": [
                          {
                            "bool": {
                              "should": [
                                {
                                  "match_phrase": {
                                    "field3": "match_field3"
                                  }
                                }
                              ],
                              "minimum_should_match": 1
                            }
                          },
                          {
                            "bool": {
                              "filter": [
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "match_phrase": {
                                          "field4": "match_field4"
                                        }
                                      }
                                    ],
                                    "minimum_should_match": 1
                                  }
                                },
                                {
                                  "bool": {
                                    "should": [
                                      {
                                        "match_phrase": {
                                          "field5": "match_field5"
                                        }
                                      }
                                    ],
                                    "minimum_should_match": 1
                                  }
                                }
                              ]
                            }
                          }
                        ]
                      }
                    }
                  ]
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-05-04T15:14:08.772Z",
              "lte": "2022-08-04T15:14:08.772Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },

Thank you in advance if you take the time to help me.

[flash1293 (Joe Reuter)]

The difference between filter and must is that must clauses contribute to the score - as hits are usually ordered by time not by score in Discover and calculating the score can be expensive depending on the situation, filter is used.

For doing an "or" query, you need to do a should clause with minimum_should_match: 1, then add all the various possibilities to the same should array.

A:1 or B:2 becomes

"bool": {
            "should": [
              {
                "bool": {
                  "should": [
                    {
                      "match": {
                        "A": "1"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              },
              {
                "bool": {
                  "should": [
                    {
                      "match": {
                        "B": "2"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ],
            "minimum_should_match": 1
          }

[Vincent92 (vincent)]

Ok thanks it's clearer now.

But then I got another questoin, if a sequence of OR operators must be put at the same level in the should[] object, why in Kibana the following query:
field1: "match_field1" OR field2: "match_field2" OR field3: "match_field3" gives this query

"query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "should": [
              {
                "bool": {
                  "should": [
                    {
                      "match_phrase": {
                        "field1": "match_field1"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              },
              {
                "bool": {
                  "should": [
                    {
                      "bool": {
                        "should": [
                          {
                            "match_phrase": {
                              "field2": "match_field2"
                            }
                          }
                        ],
                        "minimum_should_match": 1
                      }
                    },
                    {
                      "bool": {
                        "should": [
                          {
                            "match_phrase": {
                              "field3": "match_field3"
                            }
                          }
                        ],
                        "minimum_should_match": 1
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              }
            ],
            "minimum_should_match": 1
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-05-04T15:52:56.796Z",
              "lte": "2022-08-04T15:52:56.796Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },

Instead of having a query with all terms at the same level in the should[] like this:
(I removed the filter[] and put only the should[] for a better readability)

{
          "bool": {
            "should": [
              {
                "bool": {
                  "should": [
                    {
                      "match_phrase": {
                        "field1": "match_field1"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              },
              {
                "bool": {
                  "should": [
                      {
                         "match_phrase": {
                              "field2": "match_field2"
                          }
                       }
                   ],
                   "minimum_should_match": 1
                }
              },
              {
                  "bool": {
                        "should": [
                          {
                            "match_phrase": {
                              "field3": "match_field3"
                            }
                          }
                        ],
                      "minimum_should_match": 1
                    }
               }
            "minimum_should_match": 1
          }
        }

In itself it's the same since there are only should[] but I can't see the point of adding nesting levels.

Or was the example you showed me in your answer not meant to be contained in a filter[] but in a should[] object directly?

Thanks again for your quick response

[Vincent92 (vincent)]

Sorry for the second question that was quite useless, by going back to look in detail at the construction of queries in kibana I finally understood their construction.

[flash1293 (Joe Reuter)]

You are right though that the query structure created by Kibana is not as dense as it could be - it's possible to optimize further, but it won't change much.