Watcher not sending email to alert on disk space

Elastic Stack Elasticsearch
1

Hello,
I wrote a watch to send an email and make a log output if the disk space on any host goes over 90% usage. It has run a bunch of times and I know that the indexes have hosts that have over 90% usage, but the watch doesn't trigger the alert and I cannot figure out why. Any help is appreciated with figuring out where I'm going wrong. Below is my watch and the output I get when I run a GET on it.

PUT _xpack/watcher/watch/disk_space_watch
{
  "trigger" : { "schedule" : { "interval" : "1m" }},
  "input" : {
    "search" : {
      "request" : {
        "indices" : [ "filesystem*" ]
      },
	  "extract": [ "system.filesystem.used.pct", "beat.name", "system.filesystem.mount_point" ]
    }
  },
  "condition" : {
    "compare" : { "ctx.payload.system.filesystem.used.pct" : { "gte" : 0.89 }}
  },
  "actions" : {
	"email_me": {
		"email" : {
			"to" : "My@email",
			"subject" : "{{ctx.payload.beat.name}} {{ctx.payload.system.filesystem.mount_point}} {{ctx.payload.system.filesystem.used.pct}}",
			"body" : "nothing here",
			"priority" : "high"
		}
	},
	"log" : { 
		"logging" : {
			"text" : "{{ctx.payload.beat.name}} {{ctx.payload.system.filesystem.mount_point}} {{ctx.payload.system.filesystem.used.pct}}" 
    }
  }
  }
}

And here is the output:

GET _xpack/watcher/watch/disk_space_watch
{
  "found": true,
  "_id": "disk_space_watch",
  "status": {
    "state": {
      "active": true,
      "timestamp": "2018-06-12T20:54:24.475Z"
    },
    "last_checked": "2018-06-13T13:27:00.230Z",
    "actions": {
      "email_me": {
        "ack": {
          "timestamp": "2018-06-12T20:54:24.475Z",
          "state": "awaits_successful_execution"
        }
      },
      "log": {
        "ack": {
          "timestamp": "2018-06-12T20:54:24.475Z",
          "state": "awaits_successful_execution"
        }
      }
    },
    "execution_state": "execution_not_needed",
    "version": 1216
  },
  "watch": {
    "trigger": {
      "schedule": {
        "interval": "1m"
      }
    },
    "input": {
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "filesystem*"
          ],
          "types": []
        }
      }
    },
    "condition": {
      "compare": {
        "ctx.payload.system.filesystem.used.pct": {
          "gt": 0.9
        }
      }
    },
    "actions": {
      "email_me": {
        "email": {
          "profile": "standard",
          "priority": "high",
          "to": [
            "my@email"
          ],
          "subject": "{{ctx.payload.beat.name}} {{ctx.payload.system.filesystem.mount_point}} {{ctx.payload.system.filesystem.used.pct}}",
          "body": {
            "text": "nothing here"
          }
        }
      },
      "log": {
        "logging": {
          "level": "info",
          "text": "{{ctx.payload.beat.name}} {{ctx.payload.system.filesystem.mount_point}} {{ctx.payload.system.filesystem.used.pct}}"
        }
      }
    }
  }
}
2

What is the output when you simulate the watch? That's what you need to look at to first check if the input query is actually loading anything into the context, and then to check you have the paths to the values in the context defined correctly in your condition.

3

I think I may be looking at this the wrong way. Is it possible to have it send an email for each host that is alerting, or can I only send an email that is just a big list?

I want to be able to have it send out an email for each host that comes up in that search, so that may be where my code is wrong.

5

Please don't hijack my post for an unrelated topic. Just create a new topic.

6

can you please include the output of the execute watch API here, otherwise debugging will be super hard for anyone.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.