Alert results are showing incorrect

Hi Team,

I have created a watcher to detect for failed status in a particular index, attached the screenshot for reference.
the watcher seems to be working fine, but when i have observed care fully the results are quite wrong.
in the discover section i could see 2 instances of failed keyword in last 24hrs but in my watcher simulate i am getting the count as 6.
Can any one please shed some light what i am doing wrong.
if i remove this piece of code "{ "range" : { "@timestamp" : { "from": "now-1d", "to": "now" } } }," from watcher i get the whole result as 93 which matches with my discover section.
watcher code :

  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "logs-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "size": 0,
          "query" : {
          "bool" : {
          "filter" : [
        { "range" : { "@timestamp" : {  "from": "now-1d", "to": "now"  } } },
        { "match" : { "Status.keyword": "Failed"  }}
    ]
  }
}
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 1
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          ""
        ],
        "subject": "Watcher Notification",
        "body": {
          "text": "{{ctx.payload.hits.total}}"
        }
      }
    }
  }
}

Simulate result --> removed the top lines

  "result": {
    "execution_time": "2022-03-02T16:07:48.513Z",
    "execution_duration": 16,
    "input": {
      "type": "search",
      "status": "success",
      "payload": {
        "_shards": {
          "total": 30,
          "failed": 0,
          "successful": 30,
          "skipped": 0
        },
        "hits": {
          "hits": [],
          "total": 6,
          "max_score": null
        },
        "took": 15,
        "timed_out": false
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "logs-*"
          ],
          "rest_total_hits_as_int": true,
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "filter": [
                  {
                    "range": {
                      "@timestamp": {
                        "from": "now-1d",
                        "to": "now"
                      }
                    }
                  },
                  {
                    "match": {
                      "Status.keyword": "Failed"
                    }
                  }
                ]
              }
            }
          }
        }
      }
    },
    "condition": {
      "type": "compare",
      "status": "success",
      "met": true,
      "compare": {
        "resolved_values": {
          "ctx.payload.hits.total": 6
        }
      }
    },
    "actions": [
      {
        "id": "send_email",
        "type": "email",
        "status": "simulated",
        "email": {
          "message": {
            "id": "send_email__inlined__5b16ca04-0632-427d-8c13-2c9c132ec8b0-2022-03-02T16:07:48.513492828Z_27",
            "sent_date": "2022-03-02T16:07:48.530859562Z",
            "to": [
              "xxxx@gmail.com"
            ],
            "subject": "Watcher Notification",
            "body": {
              "text": "6 "
            }
          }
        }
      }
    ]
  },
  "messages": []
}

The query itself looks ok, one thing that stands out. You are using a match query, so the Failed term will not get analyzed when running against a keyword field.Is it possible, that you have documents with Failed and failed as values and this is causing a discrepancy?

Apologies @spinscale Didn't get this above point ?
also i have crosschecked and i could find only 3-4 status (Completed/Queued/Failed/-)
but if i remove the timestamp range the result is matched with the total no of failures

Can you share the timestamps of documents that are supposed to match but do not?

Your sample screenshot is only checking 500 docs, but not the whole dataset within a time range, so this cannot be trusted without running exact queries (that is part of Kibana's discover functionality).

One more thing to check: Judging from the found shards in the watcher response, 30 shards are hit. Does the same apply, when you the exact same query within dev tools as the elastic adminuser?

sharing the Timestamps of the documents below
yes I have observed that the watcher result matches when i apply the same query in dev tools, currently the value is 7 but in discover it shows me only 4 records.
also i did a visualization in a pie chart for completed and failed status here the results are different when compared to discover and watcher .


@spinscale can you please help

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.