# Include Fields in Watcher Email Alert

**URL:** <https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [January 16, 2018, 11:46am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700 "2018-01-16T11:46:11Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [January 16, 2018, 11:46am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/1 "2018-01-16T11:46:12Z")

</div>

Hi there,

I have set up an advanced Watch and it triggers upon 3 or more of the same TargetUserName with the EventID:4625, basically that alerts on any user failing to logon 3 times in 3 minutes. Here is my body section of the email callback:

```
"body": {
          "text": "The user: {{ctx.payload.TargetUserName}} failed to logon: {{ctx.payload.hits.total}} times."
        }

```

The second ctx call works and I am shown the amount of times the TargetUserName field appeard, but as for the: {{ctx.payload.TargetUserName}} no data shows, I presume I am doing something wrong here?

Cheers,

G

Below is the entire Watch, just in case anyone needs it.

```
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m",
                      "lte": "now"
                    }
                  }
                },
                {
                  "term": {
                    "EventID": "4625"
                  }
                }
              ]
            }
          },
          "aggs": {
            "User": {
              "terms": {
                "field": "TargetUserName.keyword"
              }
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.User.buckets.0.doc_count": {
        "gte": 3
      }
    }
  },
  "actions": {
    "email_admin": {
      "throttle_period_in_millis": 50000,
      "email": {
        "profile": "standard",
        "to": [
          "xxxxxxxxxxxx@mycompany.com"
        ],
        "subject": "Warning: Multiple Failed Logons",
        "body": {
          "text": "The user: {{ctx.payload.TargetUserName}} failed to logon: {{ctx.payload.hits.total}} times."
        }
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 16, 2018, 11:51am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/2 "2018-01-16T11:51:03Z")

</div>

can you please include the output of the execute watch API as well? Thanks.

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [January 16, 2018, 1:29pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/3 "2018-01-16T13:29:35Z")

</div>

Hi @spinscale could you instruct me on where to find the output of the execute watch API?

Cheers,

G

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 16, 2018, 1:41pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/4 "2018-01-16T13:41:29Z")

</div>

Hey,

it is a dedicated API that captures the whole watch execution output and simplifies debugging tremendously.

See the docs at [https://www.elastic.co/guide/en/elasticsearch/reference/6.1/watcher-api-execute-watch.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.1/watcher-api-execute-watch.html)

--Alex

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [January 16, 2018, 2:09pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/5 "2018-01-16T14:09:17Z")

</div>

Here you are:

```
{
  "_id": "W1234_b2b77b91-c0f7-4ee2-9970-3aa41112bc59-2018-01-16T14:06:36.040Z",
  "watch_record": {
    "watch_id": "W1234",
    "node": "uy-aQe2PT12NAjCj-B7QKQ",
    "state": "executed",
    "status": {
      "state": {
        "active": true,
        "timestamp": "2018-01-16T14:04:36.002Z"
      },
      "last_checked": "2018-01-16T14:06:36.040Z",
      "last_met_condition": "2018-01-16T14:06:36.040Z",
      "actions": {
        "email_admin": {
          "ack": {
            "timestamp": "2018-01-16T14:05:36.403Z",
            "state": "ackable"
          },
          "last_execution": {
            "timestamp": "2018-01-16T14:06:36.040Z",
            "successful": true
          },
          "last_successful_execution": {
            "timestamp": "2018-01-16T14:06:36.040Z",
            "successful": true
          }
        }
      },
      "execution_state": "executed",
      "version": 23129
    },
    "trigger_event": {
      "type": "manual",
      "triggered_time": "2018-01-16T14:06:36.040Z",
      "manual": {
        "schedule": {
          "scheduled_time": "2018-01-16T14:06:36.040Z"
        }
      }
    },
    "input": {
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "*"
          ],
          "types": [],
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "filter": [
                  {
                    "range": {
                      "@timestamp": {
                        "gte": "now-5m",
                        "lte": "now"
                      }
                    }
                  },
                  {
                    "term": {
                      "EventID": "4625"
                    }
                  }
                ]
              }
            },
            "aggs": {
              "User": {
                "terms": {
                  "field": "TargetUserName.keyword"
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "compare": {
        "ctx.payload.aggregations.User.buckets.0.doc_count": {
          "gte": 3
        }
      }
    },
    "metadata": {
      "name": "Failed User Logons",
      "xpack": {
        "type": "json"
      }
    },
    "result": {
      "execution_time": "2018-01-16T14:06:36.040Z",
      "execution_duration": 2347,
      "input": {
        "type": "search",
        "status": "success",
        "payload": {
          "_shards": {
            "total": 122,
            "failed": 0,
            "successful": 122,
            "skipped": 0
          },
          "hits": {
            "hits": [],
            "total": 38,
            "max_score": 0
          },
          "took": 12,
          "timed_out": false,
          "aggregations": {
            "User": {
              "doc_count_error_upper_bound": 0,
              "sum_other_doc_count": 0,
              "buckets": [
                {
                  "doc_count": 38,
                  "key": "MyNameGoesHere"
                }
              ]
            }
          }
        },
        "search": {
          "request": {
            "search_type": "query_then_fetch",
            "indices": [
              "*"
            ],
            "types": [],
            "body": {
              "size": 0,
              "query": {
                "bool": {
                  "filter": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-5m",
                          "lte": "now"
                        }
                      }
                    },
                    {
                      "term": {
                        "EventID": "4625"
                      }
                    }
                  ]
                }
              },
              "aggs": {
                "User": {
                  "terms": {
                    "field": "TargetUserName.keyword"
                  }
                }
              }
            }
          }
        }
      },
      "condition": {
        "type": "compare",
        "status": "success",
        "met": true,
        "compare": {
          "resolved_values": {
            "ctx.payload.aggregations.User.buckets.0.doc_count": 38
          }
        }
      },
      "actions": [
        {
          "id": "email_admin",
          "type": "email",
          "status": "success",
          "email": {
            "account": "outlook_account",
            "message": {
              "id": "W1234_b2b77b91-c0f7-4ee2-9970-3aa41112bc59-2018-01-16T14:06:36.040Z",
              "sent_date": "2018-01-16T14:06:36.054Z",
              "to": [
                "yournamegoeshere@myawesomecompany.com"
              ],
              "subject": "Warning: Multiple Failed Logons",
              "body": {
                "text": "The user: failed to logon: 38 times."
              }
            }
          }
        }
      ]
    },
    "messages": []
  }
}

```

Also please not, I have changed the: `The user: {{ctx.payload.TargetUserName}} failed to logon` to  
`The user: {{ctx.payload.TargetUserName.keyword}} failed to logon` but that didn't work.

If you need any more info please do no hesitate to ask 🙂

Cheers,

G

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 16, 2018, 3:55pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/6 "2018-01-16T15:55:47Z")

</div>

Check out the `result.input.payload` object from the execute watch API. It does not contain the any field named `TargetUserName` and thus is why that is empty.

You can only access fields from that payload in your mustache templates or your scripts.

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [January 16, 2018, 4:11pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/7 "2018-01-16T16:11:13Z")

</div>

How would I go about including TargetUserName.keyword?

Cheers,

G

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 17, 2018, 9:41am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/8 "2018-01-17T09:41:46Z")

</div>

judging from your aggregation you want to include `ctx.payload.aggregations.User.buckets` and every key from there.

Try this in your email body as a starter

```auto
{{#ctx.payload.aggregations.User.buckets}}{{key}}{{/ctx.payload.aggregations.User.buckets}}

```

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [January 17, 2018, 1:47pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/9 "2018-01-17T13:47:03Z")

</div>

@spinscale Hi, thank you for that, it worked! I'm not really sure how I would have got there by myself, this seems quite confusing. Could you explain how I could have found: `{{#ctx.payload.aggregations.User.buckets}}{{key}}{{/ctx.payload.aggregations.User.buckets}}` on my own?

Cheers,

G

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 18, 2018, 8:32am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/10 "2018-01-18T08:32:18Z")

</div>

Hey,

it's actually no rocket science. One of the reason I was asking so hard for the execute watch API output is the fact, that you can see in the `input.result.payload` field the whole data structure that you are able to access.

As you can see, there is no `TargetUserName.keyword`.

This blog post might help to establish a solid workflow when writing and developing watches:

> **[Watching the watches: Writing, debugging and testing watches
	  	 | Elastic](https://www.elastic.co/blog/watching-the-watches-writing-debugging-and-testing-watches)**
>
> TLDR; This blog post will outline how to efficiently write, execute, and debug watches. This will be useful if you are writing your own watches or if you have to debug watches written by your colleagu...

--Alex

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [January 18, 2018, 2:04pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/11 "2018-01-18T14:04:50Z")

</div>

@spinscale Just read through that and it seems really useful, so thanks!

I believe that I need to use the extract function to saves certain fields when the query if run. I will post below the watch I now have, I have the issue of no know where to put the extract function.

```
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "types": [],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "filter": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-1m",
                      "lte": "now"
                    }
                  }
                },
                {
                  "term": {
                    "EventID": [
                      "4728",
                      "4732",
                      "4756"
                    ]
                  }
                }
              ]
            }
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.User.buckets.0.doc_count": {
        "gte": 1
      }
    }
  },
  "actions": {
    "email_admin": {
      "throttle_period_in_millis": 50000,
      "email": {
        "profile": "standard",
        "to": [
          "email@email.com"
        ],
        "subject": "Warning: User Added to group",
        "body": {
          "text": "A user has been added to the following group: {{ctx.payload.TargetUserName}}"
        }
      }
    }
  }
} 

```

I would like to extract the TargetUserName field, I am unsure where to add:  
`"extract": ["TargetUserName"]`

Please could you provide some guidance? I feel that I am almost there with these watches!

Cheers,

G

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 18, 2018, 2:21pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/12 "2018-01-18T14:21:59Z")

</div>

Hey,

using extract is purely optional. All it does is reducing the payload, but you already have everything available.

if you want to use it, see [https://www.elastic.co/guide/en/x-pack/6.1/input-search.html#\_extracting\_specific\_fields](https://www.elastic.co/guide/en/x-pack/6.1/input-search.html#_extracting_specific_fields)

--Alex

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [January 19, 2018, 3:45pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/13 "2018-01-19T15:45:04Z")

</div>

@spinscale I have read through that and am still confused. I don't understand why when I execute the watch, there are no fields returned. I would have thought it the watch runs the query:  
"EventID = 4625" then it would return the data from the logs that include this EventID?

Cheers,

G

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 19, 2018, 3:46pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/14 "2018-01-19T15:46:21Z")

</div>

you explicitely configured `size: 0` in your query, so you are not getting back any hits

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [January 19, 2018, 3:49pm UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/15 "2018-01-19T15:49:47Z")

</div>

Oh okay, I had just copied a template watch and worked from there. I think this was where I was going wrong. I seem to have everything else pretty much sorted watch wise, I just needed to look at returning the values from the query.

Thank you so much for all the help you have given me, it is much appreciated.

Cheers,

G

---

<div class="post-metadata">

**Author:** ![co40](https://avatars.discourse-cdn.com/v4/letter/c/df705f/32.png) [@co40](https://discuss.elastic.co/u/co40)\
**Post date:** [January 30, 2018, 12:57am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/16 "2018-01-30T00:57:45Z")

</div>

Maybe this needs to be separate entries.

What's the best way to include specific fields from all ctx.payload.hits.hits in a tabular format within email boxy? Making it easy to read for end-users.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 30, 2018, 7:43am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/17 "2018-01-30T07:43:08Z")

</div>

as that is a completely new topic, making it separate makes a ton of sense.

You could use an html body in an email and make use of a table.

You can loop through the hits like this

```auto
{#ctx.payload.hits.hits}* {_source.my_field}{/ctx.payload.hits.hits}

```

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [February 5, 2018, 10:42am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/18 "2018-02-05T10:42:27Z")

</div>

Hi @spinscale,

I have had chance to get look into advanced watches again.

I am now getting values returned after looking at your advice.

```
"hits": {
          "hits": [
            {
              "_index": "main",
              "_type": "doc",
              "_source": {
                "process_id": 620,
                "computer_name": "TEST",
                "keywords": [
                  "Audit Failure"
                ],
                "log_name": "Security",
                "level": "Information",
                "record_number": "57628",
                "event_data": {
                  "Status": "0xc000006d",
                  "ProcessName": "-",
                  "LogonType": "3",
                  "IpPort": "0",
                  "SubjectLogonId": "0x0",
                  "TransmittedServices": "-",
                  "KeyLength": "0",
                  "LmPackageName": "-",
                  "SubjectUserName": "-",
                  "FailureReason": "%%2313",
                  "WorkstationName": "TEST",
                  "SubjectDomainName": "-",
                  "IpAddress": "TESTIP",
                  "ProcessId": "0x0",
                  "SubStatus": "0xc000006a",
                  "TargetDomainName": "TEST",
                  "LogonProcessName": "NtLmSsp ",
                  "SubjectUserSid": "S-1-0-0",
                  "TargetUserSid": "S-1-0-0",
                  "AuthenticationPackageName": "NTLM"
                },
                "opcode": "Info",
                "tags": [
                  "beats_input_codec_plain_applied"
                ],
                "thread_id": 496,
                "@timestamp": "2018-02-05T10:33:50.722Z",
                "task": "Logon",
                "event_id": 4625,
                "provider_guid": "{54849625-5478-4994-A5BA-3E3B0328C30D}",
                "activity_id": "{2BF93E71-95E4-0001-8B3E-F92BE495D301}",
                "beat": {
                  "hostname": "TEST",
                  "name": "TEST",
                  "version": "5.6.3"
                },
                "host": "TEST",
                "target_username": "TestUser",
                "source_name": "Microsoft-Windows-Security-Auditing"

```

Above is the output from the watch query, I have set up the email body:

```
    "actions": {
"email_admin": {
  "throttle_period_in_millis": 50000,
  "email": {
    "profile": "standard",
    "to": [
      "george.townson@commissum.com"
    ],
    "subject": "Warning: User Added to Security Enabled Group",
    "body": {
      "text": "A user was added to a security enabled group. {{#ctx.payload.hits.hits}} {{_source.name.target_username}} {{/ctx.payload.hits.hits}}"
    }
  }
}

```

}

Currently I see: "A user was added to a security enabled group. " as the output. How would I get the email callback to return every "target\_username", so I can see which users failed to login?

cheers,

G

---

<div class="post-metadata">

**Author:** ![MultiplierMultiplier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/multipliermultiplier/32/25063_2.png) [@MultiplierMultiplier](https://discuss.elastic.co/u/MultiplierMultiplier)\
**Post date:** [February 5, 2018, 10:59am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/19 "2018-02-05T10:59:35Z")

</div>

I've just worked it out. Looked back at your example and realized I had 'name.' in the section of code that returns my 'target\_username' field.

Original non working:

```
    "subject": "Warning: User Added to Security Enabled Group",
        "body": {
          "text": "A user was added to a security enabled group. {{#ctx.payload.hits.hits}} {{_source.name.target_username}} {{/ctx.payload.hits.hits}}"
        }

```

New code now working:

```
"subject": "Warning: User Added to Security Enabled Group",
    "body": {
      "text": "A user was added to a security enabled group. {{#ctx.payload.hits.hits}} {{_source.target_username}} {{/ctx.payload.hits.hits}}"
    }

```

Hope this helps someone,

Cheers,

George

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 5, 2018, 11:00am UTC](https://discuss.elastic.co/t/include-fields-in-watcher-email-alert/115700/20 "2018-03-05T11:00:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.