How do I setup watcher to only alert on new messages?

alerting

(Dennis) #1

I've been working through setting up watcher to alert on events it finds in the logs. I set up a watcher like below and every 20 seconds (in the example) it will send an email if it finds a match. But it won't stop. It will keep sending emails every 20 seconds. So, as far as I can tell, you get two options with watcher:

  1. Use throttling to set a time on the length of time between emails
  2. Add an /_ack? to the watch and it will ignore future matches until the condition becomes "ok" and then it will send emails the next time it see the match (sort of what i want)

I need and option 3 and it's probably possible to set it up. I'm just not sure how to yet.

  1. Alert/email on matches in the logs. Then stop sending emails because there is nothing new (ignoring the ones it's already found). Then if you get a new event, send another email but don't include the previous matches. Is this possible?

This is my watch:

 curl -XPUT 'http://localhost:9200/_watcher/watch/my_event_to_match' -d '{
  "trigger" : {
    "schedule" : { "interval" : "20s" }
  },
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "logstash-2015.08.13" ],
        "body" : {
          "query" : {
            "match" : { "message": "my_event_to_match" }
          }
        }
      }
    }
  },
  "condition" : {
    "compare" : { "ctx.payload.hits.total" : { "gt" : 0 }} #<-- do I need this condition?
  },
  "actions" : {
    "send_email" : {
      "email" : {
        "to" : "me@me.com ",
        "subject" : "my_event_to_match Warning from Watcher",
        "body" : "my_event_to_match MESSAGE: Please see attached",
        "attach_data" : true
      }
    }
  }
}'

This is what I'm using to implement the acknowledgement:

curl -XPUT http://localhost:9200/_watcher/watch/my_event_to_match/_ack?

I also use this line to list out all the watches I'm currently running:

curl -s -XGET 'http://localhost:9200/.watches/_search' -d {} | python -m json.tool | grep _id

Can anyone give me some pointers on setting up the watch to only alert on "new" events only and ignore ones it's already found.

Thanks in advance.

Dennis


Watcher time filter problem
(Steve Kearns) #2

Hi Dennis,

Your query is currently setup to search across all time, so once you have one record in the system that matches, you will get a match every time the query executes. You can use simple date math in the query portion to filter just to recent documents.

Here's a slightly modified watch I used in the recent Watcher Webinar. Note the way the query section is written - I filter the results to just the results that occurred in the last 25 seconds. In this case, I setup a slightly larger time range for my query than I do for my trigger interval, just to make sure I don't miss anything that might have been in-flight on it's way into ES.

PUT _watcher/watch/twitter_watcher_mention
{
  "trigger": {
    "schedule": {
      "interval": "20s"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [ "logstash-*" ],
        "body": {
          "query": {
            "filtered": {
              "query": {
                "match_phrase": {"message": "my_event_to_match"}
              },
              "filter": {
                "bool": {
                  "must": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-25s"
                        }
                      }
                    }
                  ]
                }
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "log" : {
       "logging" : {
         "text" : "There were {{ctx.payload.hits.total}} Tweets at {{ctx.execution_time}}"
       }
     }
  }
}

Stop Watcher after first alert
(Dennis) #3

@skearns - that's working now. Thanks for sharing the code.

Just one more quick question, when it comes to doing matche or match_phrase (with some examples below), will they just work or need to be formatted in a particular way?

Examples:

[Hardware Error]
fn[lr][fr].*LINEPROTO-5-UPDOWN.*down
LINK-5-CHANGED.*reset

Are wildcards * allowed and []. Do you need to ignore with \ ?

Thanks for you help,

Dennis


(system) #4