Not Sending Unique Errors

Elastic Stack Elasticsearch
1

Hey there,

Right now, I have a watch set up to send errors within a set window to a hipchat webhook. Unfortunately, it is only sending the most recent error. I would like to send all unique errors in that window but unsure how. The following is my current curl comman:

curl -XPUT 'localhost:9200/_watcher/watch/api-error' -d '{
      "trigger" : {
          "schedule" : { "interval" : "10s" } 
        },
        "input" : {
          "search" : {
            "request" : {
              "indices" : [ "filebeat" ],
              "body" : {
                "query" : {
                  "filtered" : {
                    "query" : {
                  "match_phrase" : { "source": "/var/www/html/logs/error_log" }
                  },
                  "filter" : {
                    "bool": {
                    "must": [
                    {
                      "range": {
                        "@timestamp" : {
                        "from" : "now-5m",
                        "to" : "now"
                        }
                      }
                    }
                    ]
                  }
                }
              }
            }
          }
          }
          }
        },
        "actions" : {
        "notify-hipchat" : {
          "throttle_period" : "5m",
          "hipchat" : {
            "account" : "notify-dev-monitoring",
            "message" : {
              "body": "New error seen in api apache error_log!\n\nHost:   {{ctx.payload.hits.hits.0._source.beat.hostname}}\nMessage:   {{ctx.payload.hits.hits.0._source.message}}",
              "format" : "text",
              "color" : "red",
              "notify" : true
            }
          }
        }
      }
      }'

Any insight would be greatly appreciated!

2

Hey Matthew,

To confirm your goal - do you you want one hipchat message that contains all the hosts that had errors, or do you want to send one message per host?

The first one (one message listing multiple hosts) should be possible today, using mustache templates over the results array. So in your message body, you could do something like:

{{#ctx.payload.hits.hits}} 
  {{_source.beat.hostname}}   Message: {{ _source.message}} \n
{{/ctx.payload.hits.hits}}

I'm not able to check the syntax right this second, but that should give you the idea.

Thanks,
Steve

3

Thanks for the response Steve.

To verify, I need to show every unique message the comes through in my time frame. For example, the watch runs at 2:05 and shows 3 error messages that were logged to apache error_log between 2:00-2:05. Right now, it only shows the most recent log entry.

4

Great, then the array support in mustache seems like it should work for you!

5

So to verify, the new curl should look like this?

curl -XPUT 'localhost:9200/_watcher/watch/api-error' -d '{
      "trigger" : {
          "schedule" : { "interval" : "10s" } 
        },
        "input" : {
          "search" : {
            "request" : {
              "indices" : [ "filebeat" ],
              "body" : {
                "query" : {
                  "filtered" : {
                    "query" : {
                  "match_phrase" : { "source": "/var/www/html/logs/error_log" }
                  },
                  "filter" : {
                    "bool": {
                    "must": [
                    {
                      "range": {
                        "@timestamp" : {
                        "from" : "now-5m",
                        "to" : "now"
                        }
                      }
                    }
                    ]
                  }
                }
              }
            }
          }
          }
          }
        },
        "actions" : {
        "notify-hipchat" : {
          "throttle_period" : "5m",
          "hipchat" : {
            "account" : "notify-dev-monitoring",
            "message" : {
              "body": "New error seen in api apache error_log!\n\nHost:   {{ctx.payload.hits.hits._source.beat.hostname}}\nMessage:   {{ctx.payload.hits.hits._source.message}}",
              "format" : "text",
              "color" : "red",
              "notify" : true
            }
          }
        }
      }
      }'
6

IMO, you're going to want to iterate over the hits, so each one gets added to the message:

"body": "New error seen in api apache error_log! {{#ctx.payload.hits.hits}} \n\nHost:     {{_source.beat.hostname}}\nMessage:   {{_source.message}} {{/ctx.payload.hits.hits}}"
7

Hmm, it did not like that. Got rid of the message entirely!

8

It is actually outputting hits on a schedule without any new error. Looks blank because there wasn't an error in the timeframe. Still no cooperating as expected.

9

Ah yes, it looks like your watch doesn't have a condition, so the action will always be run, no matter what comes back from the query.

I would imagine that you could use a compare condition that looks at ctx.payload.hits.total > 0