How do I setup watcher to only alert on new messages?

Elastic Stack Elasticsearch
1

I've been working through setting up watcher to alert on events it finds in the logs. I set up a watcher like below and every 20 seconds (in the example) it will send an email if it finds a match. But it won't stop. It will keep sending emails every 20 seconds. So, as far as I can tell, you get two options with watcher:

  1. Use throttling to set a time on the length of time between emails
  2. Add an /_ack? to the watch and it will ignore future matches until the condition becomes "ok" and then it will send emails the next time it see the match (sort of what i want)

I need and option 3 and it's probably possible to set it up. I'm just not sure how to yet.

  1. Alert/email on matches in the logs. Then stop sending emails because there is nothing new (ignoring the ones it's already found). Then if you get a new event, send another email but don't include the previous matches. Is this possible?

This is my watch:

 curl -XPUT 'http://localhost:9200/_watcher/watch/my_event_to_match' -d '{
  "trigger" : {
    "schedule" : { "interval" : "20s" }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logstash-2015.08.13" ],
        "body" : {
          "query" : {
            "match" : { "message": "my_event_to_match" }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }} #<-- do I need this condition?
  },
  "actions" : {
    "send_email" : {
      "email" : {
        "to" : "me@me.com ",
        "subject" : "my_event_to_match Warning from Watcher",
        "body" : "my_event_to_match MESSAGE: Please see attached",
        "attach_data" : true
      }
    }
  }
}'

This is what I'm using to implement the acknowledgement:

curl -XPUT http://localhost:9200/_watcher/watch/my_event_to_match/_ack?

I also use this line to list out all the watches I'm currently running:

curl -s -XGET 'http://localhost:9200/.watches/_search' -d {} | python -m json.tool | grep _id

Can anyone give me some pointers on setting up the watch to only alert on "new" events only and ignore ones it's already found.

Thanks in advance.

Dennis

1 Like
2

Hi Dennis,

Your query is currently setup to search across all time, so once you have one record in the system that matches, you will get a match every time the query executes. You can use simple date math in the query portion to filter just to recent documents.

Here's a slightly modified watch I used in the recent Watcher Webinar. Note the way the query section is written - I filter the results to just the results that occurred in the last 25 seconds. In this case, I setup a slightly larger time range for my query than I do for my trigger interval, just to make sure I don't miss anything that might have been in-flight on it's way into ES.

PUT _watcher/watch/twitter_watcher_mention
{
  "trigger": {
    "schedule": {
      "interval": "20s"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [ "logstash-*" ],
        "body": {
          "query": {
            "filtered": {
              "query": {
                "match_phrase": {"message": "my_event_to_match"}
              },
              "filter": {
                "bool": {
                  "must": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-25s"
                        }
                      }
                    }
                  ]
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "log" : {
       "logging" : {
         "text" : "There were {{ctx.payload.hits.total}} Tweets at {{ctx.execution_time}}"
       }
     }
  }
}
2 Likes
3

@skearns - that's working now. Thanks for sharing the code.

Just one more quick question, when it comes to doing matche or match_phrase (with some examples below), will they just work or need to be formatted in a particular way?

Examples:

[Hardware Error]
fn[lr][fr].*LINEPROTO-5-UPDOWN.*down
LINK-5-CHANGED.*reset

Are wildcards * allowed and []. Do you need to ignore with \ ?

Thanks for you help,

Dennis