Replacing missing query

Elastic Stack Elasticsearch
1

I recently updated to the newer version of Elasticsearch and realized that the missing query is missing. I understand that you could use must_not with exists. But for my particular scenario it isn't working. I have the following configuration:

filtered = {
 function_score: {
      functions: [functions],
      query: {
           bool: {
                 must: [must have features],
                 must_not: [must not have features],
                 filter: filter,
            },
       },
  }
};

and the filter array looks like the following:

[
     ...
     {
           bool: {
                should: [
                     { missing: { field: "test" } },
                     {
                          script: {
                               script: "doc['id'].value ==  doc['test'].value"
                          }
                     },
                ]
           }
     }
     ...
]

Basically saying either field test should be missing or it should be equal to the id of the same document.

This was working fine but now I don't know how to replace this with exists query. I have tried the following:

[
     ...
     {
           bool: {
                should: [
                     {
                          bool: {
                               must_not: [
                                    { exists: {field: "test"} }
                               ]
                          }
                     },
                     {
                          script: {
                               script: "doc['id'].value ==  doc['test'].value"
                          }
                     },
                ]
           }
     }
     ...
]

This however completely ignores the script query and returns documents only if the test field is missing.

2

Hi,

I tried to recreate your problem using Elasticsearch 5.1.1 and build a minimal example hoping to reproduce the issue. However, everything seems to work as expected for me when I do it like this:

PUT /i
{  
  "mappings": {
    "t": {
      "properties": {
        "id": {
          "type": "keyword"
        },
        "test" : {
          "type" : "keyword"
        }
      }
    }
  }
}

^ Note: adding the mapping because the script needs Fielddata, which is disabled on text fields by default

PUT /i/t/1
{
  "id" : "aaa",
  "test" : "aaa"
}

PUT /i/t/2
{
  "id" : "bbb"
}

PUT /i/t/3
{
  "id" : "ccc",
  "test" : "aaa"
}

GET /i/t/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "bool": {
            "should": [
              {
                "bool": {
                  "must_not": [
                    {
                      "exists": {
                        "field": "test"
                      }
                    }
                  ]
                }
              },
               {
                "script": {
                  "script": "doc['id'].value ==  doc['test'].value"
                }
              }
            ]
          }
        }
      ]
    }
  }
}

The Output is:

"hits": [
      {
        "_index": "i",
        "_type": "t",
        "_id": "2",
        "_score": 0,
        "_source": {
          "id": "bbb"
        }
      },
      {
        "_index": "i",
        "_type": "t",
        "_id": "1",
        "_score": 0,
        "_source": {
          "id": "aaa",
          "test": "aaa"
        }
      }
    ]

I think this is what you expected. I suspect there is something else going on here that is not related to the replacement of the "missing" query alone. I'd suggest taking out parts of your query independently and testing them on a very small dataset so debug this further.

3

Hey! Thanks! The problem was the mapping. Earlier I tried "script": "doc['id'].value == doc['test'].value" and "script": "doc['id'].keyword.value == doc['test'].keyword.value" and both weren't working. This is solved after mapping the field correctly.

Without the mapping, the script query is ignored I believe.

4

Great to hear, just for reference and if others are reading this thread: which version of ES are you using? I got errors when trying to run the script part on a "text" field in 5.1.1.

5

I am currently on 5.0.1. I don't get errors though. It is simply ignored and the bool>must_not>exists query above it takes over.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.