Replacing filtered and or filter query with bool (Elasticsearch 5.2)

[lukejolly]

I upgraded to Elasticsearch 5.2 and this query no longer works:

POST _search
{
  "size":3000,
  "query":{
    "filtered":{
      "query":{
        "range":{
          "@timestamp":{"gt":"now-2m", "lte":"now"}
        }
      },
      "filter":{
         "or":[
          {"term":{"severity":"ERROR"}},
          {"term":{"severity":"FATAL"}}
        ]
      }
    }
  }
}

It resulted in this error:

{"error":{"root_cause":[{"type":"parsing_exception","reason":"no [query] registered for [filtered]","line":1,"col":42}],"type":"parsing_exception","reason":"no [query] registered for [filtered]","line":1,"col":42},"status":400}

After switching to using bool and must and leaving the filter or I got this error:

{"error":{"root_cause":[{"type":"parsing_exception","reason":"[or] query malformed, no start_object after query name","line":1,"col":168}],"type":"parsing_exception","reason":"[or] query malformed, no start_object after query name","line":1,"col":168},"status":400}

I thought this would work, but it's returning no results when it should:

POST _search
{
  "size":3000,
  "query":{
    "bool":{
      "must":{
        "range":{
          "@timestamp":{"gt":"now-2m", "lte":"now"}
        }
      },
      "should":[
          {"term":{"severity":"ERROR"}},
          {"term":{"severity":"FATAL"}}
      ],
      "minimum_should_match": 1
    }
  }
}

What am I missing?

[cbuescher (Christoph)]

Hi,

the query looks okay, the problem might have to do with changes on the default mapping for string fields. When I tried reproducing this I directly indexed documents like this without defining a mapping:

PUT /test/type/1
{
  "@timestamp" : "2017-03-14",
  "severity" : "FATAL"
}

The mapping this creates in 5.2 looks like this:

{
  "test": {
    "mappings": {
      "type": {
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "severity": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  }
}

This means that the severity field itself is analyzed (thus lowercased) and doesn't match the upper case "ERROR" in your term query. If you use the severity.keyword field instead it should work:

OST /test/type/_search
{
  "size":3000,
  "query":{
    "bool":{
      "must":{
        "range":{
          "@timestamp":{"gt":"now-2m", "lte":"now"}
        }
      },
      "should":[
          {"term":{"severity.keyword":"ERROR"}},
          {"term":{"severity.keyword":"FATAL"}}
      ],
      "minimum_should_match": 1
    }
  }
}

[cbuescher (Christoph)]

I forgot to mention: since you had the or part in a filter in the original query and you are not interested in scoring for this part, you should put it in the filter section of the overall bool query now. Combining the two clauses should work smothing like this:

...
"bool": {
      "must": { ... },
      "filter": {
        "bool": {
          "should": [
            {
              "term": {
                "severity.keyword": "ERROR"
              }
            },
            {
              "term": {
                "severity.keyword": "FATAL"
              }
            }
          ],
          "minimum_should_match": 1
        }
      }
    }
...

[lukejolly]

Thanks tons! This fixed that query and got me on the right track for fixing the rest of mine.