Query Help: must_not keeps matching document

Elastic Stack Elasticsearch
1

I'm trying to write a watcher alert that triggers when any other watcher alert fails to execute. However, due to a bug, we're getting a lot of failed watch executions because a given watch is executed at the same time by multiple nodes resulting in a version mismatch exception. Until this bug is fixed and we've upgraded versions, i've updated our alert to exclude watch execution failures of this nature. However, what i think should be a working query, is not excluding the failed executions. i've narrowed it down to the "must_not" clause not kicking out documents that contain the term.

Here is a document that should be excluded (cleaned up for simplicity):

{
  "_index": ".watcher-history-7-2018.03.06",
  "_type": "doc",
  "_id": "My_Failing_Test_Alert_624fd320-47a9-476c-87bb-7fa90037e3cf-2018-03-06T16:36:28.323Z",
  "_score": null,
  "_source": {
    "watch_id": "My_Failing_Test_Alert",
    "node": "hlvBERyKR5-XvzTE-Ud44g",
    "state": "failed",
    "status": {
      ... snip ...
    },
    "trigger_event": {
      "type": "schedule",
      "triggered_time": "2018-03-06T16:36:28.323Z",
      "schedule": {
        "scheduled_time": "2018-03-06T16:36:28.190Z"
      }
    },
    "input": {
      ... snip ...
    },
    "condition": {
      ... snip ...
    },
    "metadata": {
      ... snip ...
    },
    "result": {
      ... snip ...
    },
    "exception": {
      "type": "version_conflict_engine_exception",
      "reason": "[doc][My_Failing_Test_Alert]: version conflict, current version [22262] is different than the one provided [22261]",
      "index_uuid": "YR1UZaG-Rf6PEC82jg4YLg",
      "shard": "0",
      "index": ".watches",
      ... snip ...
    }
  },
  "sort": [
    1520354188323
  ]
}

Here is my explain query informing me that the document matches:

GET .watcher-history-7-2018.03.06/doc/My_Failing_Test_Alert_624fd320-47a9-476c-87bb-7fa90037e3cf-2018-03-06T16:36:28.323Z/_explain
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must_not": [
            {
              "term": {
                "exception.type": "version_conflict_engine_exception"
              }
            }
          ],
          "must": [
            {
              "term": {
                "state": "failed"
              }
            },
            {
              "range": {
                "trigger_event.triggered_time": {
                  "gte": "now-2h"
                }
              }
            }
          ]
        }
      }
    }
  }
}

And the output of the explain query:

{
  "_index": ".watcher-history-7-2018.03.06",
  "_type": "doc",
  "_id": "My_Failing_Test_Alert_624fd320-47a9-476c-87bb-7fa90037e3cf-2018-03-06T16:36:28.323Z",
  "matched": true,
  "explanation": {
    "value": 1,
    "description": "sum of:",
    "details": [
      {
        "value": 1,
        "description": "ConstantScore(+state:failed +trigger_event.triggered_time:[1520351559689 TO 9223372036854775807] -exception.type:version_conflict_engine_exception)",
        "details": []
      },
      {
        "value": 0,
        "description": "match on required clause, product of:",
        "details": [
          {
            "value": 0,
            "description": "# clause",
            "details": []
          },
          {
            "value": 1,
            "description": "DocValuesFieldExistsQuery [field=_primary_term]",
            "details": []
          }
        ]
      }
    ]
  }
}

Even if i just reduce the explain query down to just the "must_not" it still matches:

GET .watcher-history-7-2018.03.06/doc/My_Failing_Test_Alert_624fd320-47a9-476c-87bb-7fa90037e3cf-2018-03-06T16:36:28.323Z/_explain
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must_not": [
            {
              "term": {
                "exception.type": "version_conflict_engine_exception"
              }
            }
          ]
        }
      }
    }
  }
}

And the output:

{
  "_index": ".watcher-history-7-2018.03.06",
  "_type": "doc",
  "_id": "My_Failing_Test_Alert_624fd320-47a9-476c-87bb-7fa90037e3cf-2018-03-06T16:36:28.323Z",
  "matched": true,
  "explanation": {
    "value": 1,
    "description": "sum of:",
    "details": [
      {
        "value": 1,
        "description": "ConstantScore(-exception.type:version_conflict_engine_exception +*:*)",
        "details": []
      },
      {
        "value": 0,
        "description": "match on required clause, product of:",
        "details": [
          {
            "value": 0,
            "description": "# clause",
            "details": []
          },
          {
            "value": 1,
            "description": "DocValuesFieldExistsQuery [field=_primary_term]",
            "details": []
          }
        ]
      }
    ]
  }
}

The question is what am i not understanding about the query DSL thats preventing me from excluding documents that match a term? I am fairly new to querying with ES but according to this document, this should work.

https://www.elastic.co/guide/en/elasticsearch/guide/master/combining-filters.html

Thank you for your help

2

is this because the mapping created for the watcher history index has disabled indexing on the exception object?

... snip ...
"properties": {
  "condition": {
    "type": "object",
    "enabled": false
  },
  "exception": {
    "type": "object",
    "enabled": false
  },
  "input": {
    "type": "object",
    "enabled": false
  },
... snip ...

if so, is there anything in the explain output that would have informed me of this? Also, is there an appropriate way for us to modify the built in templates? Meaning, if i modify the .watch-history-7 index template, will it get overwritten? And lets say that i eventually upgrade where a new index template is made for .watch-history-8, how would i get in front of that to configure it before it gets created?

3

i guess the other way for me to deal with this is through painless looking at the data instead of relying on the querying. This would mean that i would have to do my own aggregations in code which seems possibly a bit overkill.

4

i've also tried using a script query but i for the life of me cannot get it to execute

GET .watcher-history-*/_search
{
  "query": {
    "constant_score": {
      "filter": {
        "bool": {
          "must": [
            {
              "term": {
                "state": "failed"
              }
            },
            {
              "range": {
                "trigger_event.triggered_time": {
                  "gte": "now-2h",
                  "lt": "now"
                }
              }
            },
            {
              "script": {
                "script" : {
                    "source": "!doc['exception.type'].value.equals(\"version_conflict_engine_exception\")"
                 }
              }
            }
          ]
        }
      }
    }
  },
  "sort": [
    {
      "trigger_event.triggered_time": {
        "order": "desc"
      }
    }
  ],
  "aggs": {
    "group_by_watch_id": {
      "terms": {
        "field": "watch_id"
      }
    }
  }
}

but that only throws an exception

{
  "took": 12,
  "timed_out": false,
  "_shards": {
    "total": 35,
    "successful": 34,
    "skipped": 0,
    "failed": 1,
    "failures": [
      {
        "shard": 0,
        "index": ".watcher-history-7-2018.03.06",
        "node": "Top1lr1HRBeVVr8xT95CzQ",
        "reason": {
          "type": "script_exception",
          "reason": "runtime error",
          "script_stack": [
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:81)",
            "org.elasticsearch.search.lookup.LeafDocLookup.get(LeafDocLookup.java:39)",
            """!doc['exception.type'].value.equals("version_conflict_engine_exception")""",
            "     ^---- HERE"
          ],
          "script": """!doc['exception.type'].value.equals("version_conflict_engine_exception")""",
          "lang": "painless",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "No field found for [exception.type] in mapping with types []"
          }
        }
      }
    ]
  },
  "hits": {
    "total": 0,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "group_by_watch_id": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 0,
      "buckets": []
    }
  }
}
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.