# Watcher email and slack failing

**URL:** <https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854>\
**Category:** Kibana\
**Tags:** elastic-stack-alerting\
**Created:** [March 31, 2021, 2:39am UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854 "2021-03-31T02:39:02Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![twilight](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/twilight/32/83450_2.png) [@twilight](https://discuss.elastic.co/u/twilight)\
**Post date:** [March 31, 2021, 2:39am UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/1 "2021-03-31T02:39:02Z")

</div>

Hello,

I am trying to configure slack and email action in watcher on my ES and Kibana 7.12 installed on Amazon Linux AMI.

below are my slack settings (plus webhook url in keystore):

```auto
xpack.notification.slack:
  default_account: watcher
  account:
    watcher:
      message_defaults:
        from: kibana

```

I am setting up Amazon SES email configs like this (plus password in keystore):

```auto
xpack.notification.email.account:
    ses_account:
        smtp:
            auth: true
            starttls.enable: true
            starttls.required: true
            host: email-smtp.us-east-1.amazonaws.com
            port: 587
            user: AKIAW25 ********* 53U

```

When I try to send sample email or slack message, it fails.  
I want to trace the reason in the log file but /var/log/elasticsearch is not being accessible and /var/log/elasticsearch/elasticsearch.log is not showing latest events or info, probably there are multiple log files.

How and where can I access the email / slack related error details?  
Also if there is anything I am missing in watcher action configs?

Thanks

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 31, 2021, 2:52am UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/2 "2021-03-31T02:52:48Z")

</div>

Couple questions watcher is a Gold+ license feature what level license do.you have.

Also have you considered using the new Kibana alerting framework instead of watcher?

Finally you can test watcher via the dev console / or get the last execution using the [watcher API](https://www.elastic.co/guide/en/elasticsearch/reference/current/watcher-api-get-watch.html) if I recall the failed execution / action should be in there.

You can also test it via the watcher screen in Kibana

---

<div class="post-metadata">

**Author:** ![twilight](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/twilight/32/83450_2.png) [@twilight](https://discuss.elastic.co/u/twilight)\
**Post date:** [March 31, 2021, 8:38am UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/3 "2021-03-31T08:38:16Z")

</div>

I have a 30 days trial activated and I am trying to send sample email/slack message from the Watcher screen in Kibana.  
I have been using Watcher on ES cloud stack and want to replicate the same on our self-managed stack on EC2.

My question was where can I find the Watcher logs or error details why an email or slack message is failing, or if I am missing some configs?

Thanks

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 31, 2021, 2:51pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/4 "2021-03-31T14:51:17Z")

</div>

As I mentioned above,

You can use the watcher API itself and try to execute the watcher you can look at the results and it will give you information on the error.

The errors will help you debug your config...

```auto
Create a watcher that will execute... 

PUT _watcher/watch/2db648f8-2564-4cde-8f0f-67a426f005a4
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {

Execute it..... 

POST _watcher/watch/2db648f8-2564-4cde-8f0f-67a426f005a4/_execute

Look at the Result .... 

"actions" : [
    {
      "id" : "webhook_1",
      "type" : "webhook",
      "status" : "failure",
      "error" : {
        "root_cause" : [
          {
            "type" : "unknown_host_exception",
            "reason" : "api.bvader.net: Name or service not known"
          }
        ],
        "type" : "unknown_host_exception",
        "reason" : "api.bvader.net: Name or service not known"
      }
    },

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 31, 2021, 2:54pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/5 "2021-03-31T14:54:31Z")

</div>

You can also goto the watcher screen in Kibana and look at the failed executions

 ![Screen Shot 2021-03-31 at 7.53.47 AM](https://us1.discourse-cdn.com/elastic/original/3X/8/5/858377decb990e0f14f0aa7207313f2874d52cff.png)

---

<div class="post-metadata">

**Author:** ![twilight](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/twilight/32/83450_2.png) [@twilight](https://discuss.elastic.co/u/twilight)\
**Post date:** [April 1, 2021, 2:01am UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/6 "2021-04-01T02:01:02Z")

</div>

Hello,

this is the error I saw in log:

```auto
"error": {
         ...
"caused_by": {
            "type": "s_m_t_p_send_failed_exception",
            "reason": "554 Message rejected: Email address is not verified. The following identities failed the check in region US-EAST-1: elasticsearch@ip-x-x-x-x.ec2.internal\n"
          }

```

If you see the email settings in my first post, I think I am missing the 'from' email address which I have verified with AWS SES, where do I specify that please?

---

<div class="post-metadata">

**Author:** ![twilight](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/twilight/32/83450_2.png) [@twilight](https://discuss.elastic.co/u/twilight)\
**Post date:** [April 1, 2021, 2:11am UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/7 "2021-04-01T02:11:06Z")

</div>

this is what I see related to slack action but I cannot find the clear error message or reason:

```auto
{
        "id": "slack_1",
        "type": "slack",
        "status": "failure",
        "slack": {
          "account": "watcher",
          "sent_messages": [
            {
              "status": "failure",
              "request": {
                "host": "hooks.slack.com",
                "port": -1,
                "scheme": "https",
                "method": "post",
                "headers": {
                  "Content-Type": "application/json; charset=UTF-8"
                },
                "body": "{\"username\":\"kibana\",\"text\":\"Test Alert: Watch [Disk Usage Alert] has exceeded the threshold\"}"
              },
              "response": {
                "status": 404,
                "headers": {
                  "date": [
                    "Thu, 01 Apr 2021 02:03:28 GMT"
                  ],
                  "server": [
                    "Apache"
                  ],
                  "x-envoy-upstream-service-time": [
                    "4"
                  ],
                  "transfer-encoding": [
                    "chunked"
                  ],
                  "vary": [
                    "Accept-Encoding"
                  ],
                  "x-frame-options": [
                    "SAMEORIGIN"
                  ],
                  "x-via": [
                    "envoy-www-iad-ikzh, haproxy-edge-iad-xu5y"
                  ],
                  "x-backend": [
                    "main_normal main_bedrock_normal_with_overflow main_canary_with_overflow main_bedrock_canary_with_overflow main_control_with_overflow main_bedrock_control_with_overflow"
                  ],
                  "x-slack-backend": [
                    "r"
                  ],
                  "strict-transport-security": [
                    "max-age=31536000; includeSubDomains; preload"
                  ],
                  "via": [
                    "envoy-www-iad-ikzh"
                  ],
                  "access-control-allow-origin": [
                    "*"
                  ],
                  "referrer-policy": [
                    "no-referrer"
                  ],
                  "content-type": [
                    "text/html"
                  ],
                  "x-slack-shared-secret-outcome": [
                    "shared-secret"
                  ],
                  "x-server": [
                    "slack-www-hhvm-main-iad-fu05"
                  ]
                },
                "body": "no_service"
              },
              "message": {
                "from": "kibana",
                "text": "Test Alert: Watch [Disk Usage Alert] has exceeded the threshold"
              }
            }
          ]
        }

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 1, 2021, 2:30am UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/8 "2021-04-01T02:30:22Z")

</div>

> [@twilight](#):
>
> I think I am missing the 'from' email address which I have verified with AWS SES, where do I specify that please?

Pretty sure It goes in the actual email action there is a `from` and `reply_to` see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html#email-action-attributes) not in the email setup

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 1, 2021, 2:39am UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/9 "2021-04-01T02:39:08Z")

</div>

With respect to slack since you say it is working on Elastic Cloud, perhaps simply copy the settings from the elasticsearch.yml on Elastic Cloud.

I don't recognize that error

---

<div class="post-metadata">

**Author:** ![twilight](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/twilight/32/83450_2.png) [@twilight](https://discuss.elastic.co/u/twilight)\
**Post date:** [April 1, 2021, 2:27pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/10 "2021-04-01T14:27:54Z")

</div>

> [@stephenb](#):
>
> Pretty sure It goes in the actual email action there is a `from` and `reply_to` see [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html#email-action-attributes) not in the email setup

I am trying to set a from and reply-to but the service fails to start, I have tried single, double and without quotes.

```auto
xpack.notification.email:
     from: 'Watcher <no-reply@domain.com>'
     reply-to: 'build@domain.com'
     default_account: ses_account
     account:
	ses_account:
            smtp:
                auth: true
                starttls.enable: true
                starttls.required: true
                host: email-smtp.us-east-1.amazonaws.com
                port: 587
                user: AK ***** VD53U

```

Elasticsearch service starts fine if I disable the 'from' and 'reply-to' line from .yml.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 1, 2021, 2:41pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/11 "2021-04-01T14:41:38Z")

</div>

Hi @twilight 🙂

It does not go in the account setup... per the link I sent above ([Here](https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html#email-action-attributes) again)`from` and `reply_to` go into the actual Email Action in the Watcher _ **not** _ in the account setup section in the elasticsearch.yml.

See Examples [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/watching-meetup-data.html)

```
"actions": {
    "email_me": {
      "throttle_period": "10m",
      "email": {
        "from": "<from:email address>", <! ------ HERE
        "to": "<to:email address>",
        "subject": "Open Source Events",
        "body": {
          "html": "Found events matching Open Source: <ul>{{#ctx.payload.aggregations.group_by_city.buckets}}< li>{{key}} ({{doc_count}})<ul>{{#group_by_event.buckets}}
          <li><a href=\"{{key}}\">{{get_latest.buckets.0.group_by_event_name.buckets.0.key}}</a>
          ({{doc_count}})</li>{{/group_by_event.buckets}}</ul></li>
          {{/ctx.payload.aggregations.group_by_city.buckets}}</ul>"
        }
      }
    }
  }
```

---

<div class="post-metadata">

**Author:** ![twilight](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/twilight/32/83450_2.png) [@twilight](https://discuss.elastic.co/u/twilight)\
**Post date:** [April 1, 2021, 3:04pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/12 "2021-04-01T15:04:56Z")

</div>

Thanks for the help with email action, it is fine now.

> [@stephenb](#):
>
> With respect to slack since you say it is working on Elastic Cloud, perhaps simply copy the settings from the elasticsearch.yml on Elastic Cloud.

I exactly did the same, copied from cloud .yml to local .yml.

Cloud configs:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/0/3053c2545a33968d3dfa21b48f0bf1f70d620a0d.png)

And local configs:  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/2/3221cb1fe987da0f5697ea2d86d4aad9e01327aa.png)

Webhook is stored in the same key in keystore:  
xpack.notification.slack.account.watcher.secure\_url

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 1, 2021, 3:14pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/13 "2021-04-01T15:14:14Z")

</div>

Not sure what to tell you on that perhaps.

404 is not found, sure you dont't have a typo in the slack key / value in the keystore. (It will never show in an error message). ir perhaps Network connectivity... you could try to curl the slack webhook url from the elasticsearch server

```auto
curl -X POST -H 'Content-type: application/json' --data '{"text":"Allow me to reintroduce myself!"}' YOUR_WEBHOOK_URL

```

Also you need to put that in the keystore on **EACH** elasticsearch node, unlike cloud where we do the propagation for you.

Here is my setup. I like verbose yaml to avoid mistakes.

```
xpack.notification.slack.account.monitoring.message_defaults.from: x-pack
xpack.notification.slack.account.monitoring.message_defaults.to: notifications
xpack.notification.slack.account.monitoring.message_defaults.icon: http://example.com/images/watcher-icon.jpg
xpack.notification.slack.account.monitoring.message_defaults.attachment.fallback: "X-Pack Notification"
xpack.notification.slack.account.monitoring.message_defaults.attachment.color: "#36a64f"
xpack.notification.slack.account.monitoring.message_defaults.attachment.title: "X-Pack Notification"
xpack.notification.slack.account.monitoring.message_defaults.attachment.title_link: "https://www.elastic.co/guide/en/x-pack/current/index.html"
xpack.notification.slack.account.monitoring.message_defaults.attachment.text: "One of your watches generated this notification."
xpack.notification.slack.account.monitoring.message_defaults.attachment.mrkdwn_in: "pretext, text"
```

---

<div class="post-metadata">

**Author:** ![twilight](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/twilight/32/83450_2.png) [@twilight](https://discuss.elastic.co/u/twilight)\
**Post date:** [April 1, 2021, 3:25pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/14 "2021-04-01T15:25:57Z")

</div>

> [@stephenb](#):
>
> you could try to curl the slack webhook url from the elasticsearch server

It worked with my webhook url.

Is there any way to trace what webhook url it is trying to send to? or what url is set in keystore?  
Althought the keystore list is showing a key with name 'xpack.notification.slack.account.watcher.secure\_url' but not sure about the value.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 1, 2021, 3:33pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/15 "2021-04-01T15:33:19Z")

</div>

Delete it and set it again... the value will never show anywhere... after all it IS a secret 🙂

Remove it... Restart Elasticsearch  
run the watcher it will fail.

Add it Back Carefully .... Restart Elasticsearch  
and see what you get.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 29, 2021, 3:33pm UTC](https://discuss.elastic.co/t/watcher-email-and-slack-failing/268854/16 "2021-04-29T15:33:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.