Using "lte" : "Now" in Watcher Date Range giving Unusual behaviour

[Will_Monkeys (Will Needham)]

Hey!

I'm trying to set up a Watcher that fires any time a document enter a given index, with the property Command containing the phrase "New Workset".

I got a query from the Discover tab > Inspect, then coped and pasted this into an advanced Watcher JSON like this (notice the fixed gte/lte date range):

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "should": [
              {
                "match_phrase": {
                  "Command": "New Workset"
                }
              }
            ],
            "minimum_should_match": 1
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2020-01-13T13:40:34.449Z",
              "lte": "2020-01-13T16:40:34.449Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }}

        },
        "indices": [
          "*"
        ]
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 1
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
      }
    }
  }
}

I simulated the Watcher and got 5 documents, as expected, and the Watcher fired.

Then, I wanted to change the date ranges to make them dynamic using the "now-1h" syntax. I want the Watcher to trigger if there is at least one document added to the index that meets the condition.

I thought this would be easy, but when I did it, I didn't get the results expected.

When the date is set to:

        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "now-2h",
              "lte": "now"
            }
          }
        }

I get the following, which is actually wrong (should be 5, and yes all these documents are definitely within the last 2 hours):

"condition": {
      "type": "compare",
      "status": "success",
      "met": true,
      "compare": {
        "resolved_values": {
          **"ctx.payload.hits.total": 3**
        }
      }
    },

I think I've narrowed it down to the lte:"Now" line - not functioning properly, because when I try this Date Range, I get the correct ctx.payload.hits.total of 5

          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "now-2h",
              "lte": "2020-01-13T16:40:34.449Z"
            }
          }
        }

Any ideas why this might be happening when I use lte:"Now"?

Thanks

[Will_Monkeys (Will Needham)]

I also tried changing the lte to lt, still the same error.

[Will_Monkeys (Will Needham)]

Anyone?

[Marius_Dragomir]

Maybe it's something related to the timezone of the Kibana server and the timezone of the data when it's ingested. Is there a difference that could explain this?