Watch Execution simulation not returning results

Hi,
I am trying to simulate a watch and see if the actions are triggering fine. But my problem is the search returns no results.
My query

  1. Checks for a particular index.
  2. Checks for a range
  3. Check for the servicename field to be a particular value.

This is my watch definition

    {
      "trigger": {
        "schedule": {
          "interval": "10m"
        }
      },
      "input": {
        "search": {
          "request": {
            "search_type": "query_then_fetch",
            "indices": [
              "datasolutions-svc-*"
            ],
            "body": {
              "query": {
                "bool": {
                  "filter": [
                    {
                      "term": {
                        "level": {
                          "value": "ERROR"
                        }
                      }
                    },
                    {
                      "term": {
                        "servicename": [
                          "Iit.Det.Urm.MepsSubscriber"
                        ]
                      }
                    },
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-60m"
                        }
                      }
                    }
                  ]
                }
              }
            }
          }
        }
      },
      "condition": {
        "compare": {
          "ctx.payload.hits.total": {
            "gt": 0
          }
        }
      },
      "actions": {
        "notify-slack": {
          "slack": {
            "account": "elastic_watcher_alerts",
            "proxy": {
              "host": "proxy.dom",
              "port": 80
            },
            "message": {
              "from": "Error Monitor",
              "to": [
                "#det-errors"
              ],
              "text": "The following error(s) have been logged",
              "dynamic_attachments": {
                "list_path": "ctx.payload.items",
                "attachment_template": {
                  "color": "#f00",
                  "title": "{{msg}}",
                  "title_link": "https://elastic.mid.dom:port/{{index}}/doc/{{id}}?pretty",
                  "text": "{{msg}}",
                  "fields": [
                    {
                      "title": "Server",
                      "value": "{{host}}",
                      "short": true
                    },
                    {
                      "title": "Servicename",
                      "value": "{{service}}",
                      "short": true
                    }
                  ]
                }
              }
            }
          }
        }
      },
      "transform": {
        "script": {
          "source": "['items': ctx.payload.hits.hits.collect(hit -> ['msg': hit._source.message, 'service': hit._source.servicename, 'index': hit._index, 'id' : hit._id, 'host': hit._source.agent.hostname ])]",
          "lang": "painless"
        }
      }
    }

I am trying to now test it by using the simulate option and giving it an input. This input is copied from actual data that is in the index. I copied a json document from kibana (in the discover section), so the alternate input json should be ok

Here's the alternative input

{
  "_index": "datasolutions-svc-live-7.7.0-2021.01",
  "_type": "doc",
  "_id": "Hre9SHcB1QIqYEnyxSCw",
  "_version": 1,
  "_score": null,
  "_source": {
    "exception": "System.Data.SqlClient.SqlException (0x80131904): blabla",
    "agent": {
      "hostname": "SATSVC3-DK1",
      "name": "datasolutions-svc-live",
      "id": "8c826ae1-e411-4257-a31f-08824dd58b5a",
      "type": "filebeat",
      "ephemeral_id": "e355bf8a-be67-4ed1-85f4-b9043674700e",
      "version": "7.7.0"
    },
    "log": {
      "file": {
        "path": "D:\\logs\\7DaysRetention\\Iit.Det.Urm.MepsSubscriber\\Iit.Det.Urm.MepsSubscriber.log.20210128.log"
      },
      "offset": 17754757
    },
    "level": "ERROR",
    "message": "Error while starting service.",
    "@timestamp": "2021-02-17T10:00:28.343Z",
    "ecs": {
      "version": "1.5.0"
    },
    "host": {
      "name": "datasolutions-svc-live"
    },
    "servicename": "Iit.Det.Urm.MepsSubscriber",
    "codelocation": "Iit.Det.Urm.MepsSubscriber.MepsSubscriberService.OnStart:29"
  },
  "fields": {
    "@timestamp": [
      "2021-02-17T10:00:28.343Z"
    ]
  },
  "highlight": {
    "servicename": [
      "@kibana-highlighted-field@Iit.Det.Urm.MepsSubscriber@/kibana-highlighted-field@"
    ]
  },
  "sort": [
    1611833128343
  ]
}

But when I run "simulate", I get the ctx.payload.total.hits as null because apparently it does not find any results. Result of the simulate-

    {
  "watch_id": "_inlined_",
  "node": "eMS-E34eT4-zZhGwtPNSmw",
  "state": "execution_not_needed",
  "user": "sum",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2021-02-17T10:57:04.077Z"
    },
    "last_checked": "2021-02-17T10:57:04.077Z",
    "actions": {
      "notify-slack": {
        "ack": {
          "timestamp": "2021-02-17T10:57:04.077Z",
          "state": "awaits_successful_execution"
        }
      }
    },
    "execution_state": "execution_not_needed",
    "version": -1
  },
  "trigger_event": {
    "type": "manual",
    "triggered_time": "2021-02-17T10:57:04.077Z",
    "manual": {
      "schedule": {
        "scheduled_time": "2021-02-17T10:57:04.077Z"
      }
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "datasolutions-svc-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "bool": {
              "filter": [
                {
                  "term": {
                    "level": {
                      "value": "ERROR"
                    }
                  }
                },
                {
                  "term": {
                    "servicename": [
                      "Iit.Det.Urm.MepsSubscriber"
                    ]
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-60m"
                    }
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "metadata": {
    "name": "datasolutions-svc-mepssubscriber",
    "xpack": {
      "type": "json"
    }
  },
  "result": {
    "execution_time": "2021-02-17T10:57:04.077Z",
    "execution_duration": 0,
    "input": {
      "type": "simple",
      "status": "success",
      "payload": {
        "highlight": {
          "servicename": [
            "@kibana-highlighted-field@Iit.Det.Urm.MepsSubscriber@/kibana-highlighted-field@"
          ]
        },
        "_index": "datasolutions-svc-live-7.7.0-2021.01",
        "_type": "doc",
        "_source": {
          "exception": "System.Data.SqlClient.SqlException (0x80131904): blabla",
          "agent": {
            "hostname": "SATSVC3-DK1",
            "name": "datasolutions-svc-live",
            "id": "8c826ae1-e411-4257-a31f-08824dd58b5a",
            "type": "filebeat",
            "ephemeral_id": "e355bf8a-be67-4ed1-85f4-b9043674700e",
            "version": "7.7.0"
          },
          "@timestamp": "2021-02-17T10:00:28.343Z",
          "ecs": {
            "version": "1.5.0"
          },
          "log": {
            "file": {
              "path": "D:\\logs\\7DaysRetention\\Iit.Det.Urm.MepsSubscriber\\Iit.Det.Urm.MepsSubscriber.log.20210128.log"
            },
            "offset": 17754757
          },
          "level": "ERROR",
          "host": {
            "name": "datasolutions-svc-live"
          },
          "servicename": "Iit.Det.Urm.MepsSubscriber",
          "message": "Error while starting service.",
          "codelocation": "Iit.Det.Urm.MepsSubscriber.MepsSubscriberService.OnStart:29"
        },
        "_id": "Hre9SHcB1QIqYEnyxSCw",
        "sort": [
          1611833128343
        ],
        "_score": null,
        "fields": {
          "@timestamp": [
            "2021-02-17T10:00:28.343Z"
          ]
        },
        "_version": 1
      }
    },
    "condition": {
      "type": "compare",
      "status": "success",
      "met": false,
      "compare": {
        "resolved_values": {
          "ctx.payload.hits.total": null
        }
      }
    },
    "actions": []
  },
  "messages": []
}

I am not sure what can't it find the results. Can someone tell me what is it that I am doing wrong?

I was able to solve it using the "inspect" section of discover page of the index.

Finally my input for the watcher query had to be changed to

"input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "datasolutions-svc-*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "bool": {
              "must": [],
              "filter": [
                {
                  "bool": {
                    "should": [
                      {
                        "match_phrase": {
                          "servicename": "Iit.Det.Urm.MepsSubscriber"
                        }
                      }
                    ],
                    "minimum_should_match": 1
                  }
                },
                {
                  "match_phrase": {
                    "level": "ERROR"
                  }
                },
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-10m",
                      "format": "strict_date_optional_time"
                    }
                  }
                }
              ],
              "should": [],
              "must_not": []
            }
          }
        }
      }
    }
  }

can you paste the complete output of the Execute Watch API ? check the result section of your JSON.. you will see that your search did not return any hits, so not triggering the condition sounds good to me. Also make sure the path your field is the right one, as by default in watcher 7 the rest_total_hits_as_int is set to true in order to retain BWC.

Thanks!