Watcher fire once

[Andreas_Marshall]

I am trying to setup a watcher that will fire and send email as soon as an log with "level": "error" comes in.
I could not find any build in functionality for this, so I thought I check for logs with an timestamp newer then 25 seconds ago, and run the watcher every 20 second. That would only fire on new events.

It is working if I only matches on "level": "error", but it wont fire if I include the Range:
How would I write the query to check for timestamp newer then 25 seconds ago? Or is there a better way to fire on new logs comming in?

PUT _xpack/watcher/watch/log_error_watch
{
  "trigger": {
    "schedule": {
      "interval": "20s"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          "local-test-*"
        ],
        "body": {
          "query": {
            "match": {
              "level": "Error"
            },
            "range": {
              "@timestamp": {
                "gte": "now-25s",
                "lte": "now"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "to": "test@test.com",
        "subject": "Watcher Notification",
        "body": "{{ctx.payload.hits}}",
        "priority": "highest"
      }
    }
  }
}

[spinscale (Alexander Reelsen)]

Hey,

the main question here is: how fast is your whole ingestion pipeline? If it is slower than 25 seconds, you might miss important events - even if that happens only on occasionally (you want to know especially that, because this is when your pipeline or your systems are under load).

You can use the execute watch API to see if there is any data matching your query.

Another idea would be to set a high throttle period for the watch or for the email action.

Hope that helps.

--Alex

[Andreas_Marshall]

Hi!
Thx for the answer!

We are dealing with pretty simple stuff so the ingestion pipeline wont be anywhere near 25 seconds.

But how should we use the watch API in this case? What would trigger it?

The throttle period sounds like a valid solution, but will it affect the performance in the long run?
Also, what would be the maximum number for throttle-period?

Regards
Andreas

[spinscale (Alexander Reelsen)]

Hey,

the execute watch API allows you to execute a watch for testing purposes via the REST API. See the documentation how to trigger it.

Can you explain the question about performance? I do not understand the intent behind that question.

The throttle period could be set to sth obsene high as 2000d. Another idea might be to deactivate the watch after it was run (or another watch to deactivate all the watches that have been run recently by querying the history).

--Alex

[Andreas_Marshall]

We ended up using the following without throttle-period. It is working as intended

PUT _xpack/watcher/watch/my-watch
{
  "trigger": {
    "schedule": {
      "interval": "20s"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          "test-*"
        ],
        "body": {
          "query": {
            "bool": {
              "must": {
                "match": {
                  "level": "Error"
                }
              },
              "filter": {
                "range": {
                  "@timestamp": {
                    "from": "{{ctx.trigger.scheduled_time}}||-25s",
                    "to": "{{ctx.trigger.triggered_time}}"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "to": "test@test.com",
        "subject": "Watcher Notification",
        "body": "{{ctx.payload}}",
        "priority": "highest"
      }
    }
  }
}