# Need Watcher configuration and settings ElasticSearch yml

**URL:** <https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [April 19, 2023, 9:31am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291 "2023-04-19T09:31:54Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [April 19, 2023, 9:31am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/1 "2023-04-19T09:31:54Z")

</div>

Hi Team,  
,

Anyone one Could you please help me with the watcher configuration elasticsearch yml setup as we have 13 nodes I need to add the watcher settings to send a mail alert (outlook). Challenges what am facing here. It is completely secured project and need to add watcher settings to get email alert to my client. So I referred many pages in elastic I was facing issue adding setup in Elasticsearch yml file.

Elasticsearch version 7.17 version we are using  
Totally 13 nodes ( master node - 3 , Data node - 10)

Questions — is it possible to test the watcher in one master node and one data node to enable the email alert ???

And In our environment we have default port 25 smtp

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [April 24, 2023, 5:40pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/2 "2023-04-24T17:40:00Z")

</div>

Hi @Praveen_kr,

You'll need to add the settings to all your nodes for it to start working. That can be a big ask so often I'll save those changes for when I'm going to do a rolling restart anyway. Here is the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/notification-settings.html) that should help out with watcher configuration.

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [April 25, 2023, 2:08am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/3 "2023-04-25T02:08:50Z")

</div>

Hi Sullivan,

Thanks for your support. I had configured the settings for outlook but password am missing to store in secure way. I didn’t added password in the watcher settings. How to proceed further. I need to add the password in all the nodes to move head. Till that watcher is disabled in kibana I can’t enable no option .

Could you please suggest how to add my outlook password to test the watcher email is working ??

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [April 25, 2023, 2:41am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/4 "2023-04-25T02:41:33Z")

</div>

Want to correct something from first reply, you'll need to add those changes to all the nodes, but I don't think you'll need to do a restart for the email settings. If you follow the link to the documentation provided on my original reply you'll see the following:

smtp.secure\_password is a secure and reloadable setting.  
Almost all the smtp settings are also dynamic, which means you can set them through Kibana.

As far as configuring it securely, see [here](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/secure-settings.html) for using secure settings.

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [April 25, 2023, 5:27am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/5 "2023-04-25T05:27:38Z")

</div>

Hi Sullivan,

This is my settings in yml file

Xpack.watcher.enabled: true  
Xpack.notification.email.account:  
Outlook\_account:  
Profile: outlook  
SMTP:  
auth: true  
Starttls.enable: true  
Host: [smtp-mail.outlook.com](http://smtp-mail.outlook.com)  
Port: 25  
User: [xxxxx@company.com](mailto:xxxxx@company.com)

Anything is missing in configuration please let me know.  
Password I didn’t mention but am not sure how to add here after user in the settings.

Please suggest me

Regards,  
Praveen

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [April 25, 2023, 7:25pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/6 "2023-04-25T19:25:13Z")

</div>

This is what I'd use regarding watcher in your elasticsearch.yml file:

```auto
xpack.watcher.enabled: true

```

From dev tools in kibana I'd set the following cluster settings since they are dynamic:

```auto
xpack.notification.email.account.outlook_account.profile: outlook
xpack.notification.email.account.outlook_account.smtp.auth: true
xpack.notification.email.account.outlook_account.smtp.starttls.enable: true
xpack.notification.email.account.outlook_account.smtp.host: smtp-mail.outlook.com
xpack.notification.email.account.outlook_account.smtp.user: xxxxx@company.com

```

You could totally have them display nested didn't do that here.

You will then HAVE to add the setting "xpack.notification.email.account.outlook\_account.smtp.secure\_password" to the elasticsearch keystore for things to work securely. If that isn't added there it's never going to work. Please see the following documentation about doing that: [elasticsearch-keystore docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/elasticsearch-keystore.html).

Good luck

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [April 26, 2023, 5:50am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/7 "2023-04-26T05:50:55Z")

</div>

Hi Sullivan,

Thanks a lot for ur help . Challenges what am facing adding outlook password in yml file.  
I am not sure how to add password in Keystore I went through documentation but still I have concern regarding those things.

Example : In kibana under stack management- watcher- threshold alert - action - email ——-this is not enable yet. I tried adding watcher settings in yml file on 3 master nodes and restarted the elasticsearch still I don’t see the enable in kibana. Also lot of security concern I have to go step by step as explained earlier . We have 10 data nodes and 3 master nodes . In all these nodes I need to add watcher outlook settings and restart the service ???

Please help me how will I proceed further.

Regards,  
Praveen

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [April 27, 2023, 8:01pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/8 "2023-04-27T20:01:28Z")

</div>

Please read my previous posts because I feel like I've already answered these questions, but just to be clear:

> [@Praveen\_kr](#):
>
> In all these nodes I need to add watcher outlook settings and restart the service ???

Yes, you'll need to enable watcher and do a rolling-restart.

```auto
xpack.watcher.enabled: true

```

on all the nodes (master, data, etc). Just doing some will NOT work.  
Can you tell me what your security concerns are? Without more information storing sensitive information in the elastic keystore is the best practice for handling that.

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [April 28, 2023, 12:27am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/9 "2023-04-28T00:27:30Z")

</div>

Hi Sullivan,  
My concern here adding the password in a keyword tool. Is it mandatory adding outlook password in a Keystore tool???

I don’t know how to create the Keystore from  
Beginnings and adding the outlook password?

Regards,  
Praveen

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [April 28, 2023, 4:27pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/10 "2023-04-28T16:27:30Z")

</div>

Ok, lets take a step back here and talk about what you're trying to accomplish. Reading between the lines it sounds like you might be connecting a fully functioning email address up with watcher. While that would probably work often a noreply address is used instead ([noreply@example.com](mailto:noreply@example.com)). Using an account like could limit the exposure if somehow the username and password were compromised.

Another option is if you just need some kind of alert, [watcher supports other action](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/actions.html)s besides email. For example: webhook, slack, pagerduty. Perhaps one of those would work just as well.

Finally lets talk about the options to set variables in elastic.

1. You could set the password via command line. This is not secure at all because any other user logged into the system could see the password.
2. Setting the password directly in the elasticsearch.yml file. This is a little bit better assuming you have your file permissions set correctly, but you are still dependent upon the OS properly limiting read access to the yml file.
3. Using the keystore. The keystore was made for just these purposes. It will keep secure settings encrypted until needed by the program. This is the most secure option of the three.

To answer your questions about the keystore, no it is not mandatory could use option 1 or 2 if you wanted to but those aren't better. The keystore should be made automatically by elastic and should be in the same directory as your elasticsearch.yml file.

Final thought is please read and become familiar with the links provided. The answers to many of your questions will be found in the documentation, and while I try my best to answer and help this community the ultimate responsibility for your cluster falls on you and your understanding of the issues. Having said that I'm happy to help where I can, but can only do so much.

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [May 2, 2023, 12:59pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/11 "2023-05-02T12:59:37Z")

</div>

Hi Sullivan,

I have configured watcher in yml file all 3 master nodes , but still don’t see the enable option in kibana .

I.e Kibana - stack management - watcher - create watcher - add condition - add action —— here I don’t see the email enabled.

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [May 2, 2023, 7:13pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/12 "2023-05-02T19:13:19Z")

</div>

You need to configure the watcher settings in all the nodes in the cluster. Just master nodes won't cut it. Must enable on data nodes as well. You probably won't see an enable option in kibana, but you will be able to send emails from watcher once correctly configured.

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [May 3, 2023, 2:11pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/13 "2023-05-03T14:11:19Z")

</div>

Sure Sullivan will configure same for all remaining nodes.

In Elasticsearch yml file I have mentioned all the outlook settings as per the elastic documentation. password I have created Keystore..

Question——— under the user password : can I mention $ {outlook.smtp.password} the same as below ?????

smtp: auth: true starttls.enable: true host: [smtp.office365.com](http://smtp.office365.com) port: 587 user: @outlook.com password: ${outlook.smtp.password}

Regards,  
Praveen

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [May 3, 2023, 2:50pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/14 "2023-05-03T14:50:18Z")

</div>

I believe you won't have to put the password part in the elasticsearch.yml file because it will look in the keystore for it. You will need make sure the password is in each keystore as well.

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [May 8, 2023, 1:51pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/15 "2023-05-08T13:51:53Z")

</div>

Hello Sullivan,

Appreciate your support. I am working on data nodes to add the watcher setting remaining 10 nodes. If I face any issue challenging will let you know by tomorrow.

Regards,  
Praveen

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [May 9, 2023, 10:24am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/16 "2023-05-09T10:24:54Z")

</div>

Hi Sullivan,

I have restarted all the data nodes and master nodes but the don’t see watcher enable in kibana.  
Stack management - Watcher- create threshold alert- Add action - here I don’t see enable still.  
Could you please help me out.

Also on see log message nohup.error. With 0 messages in all the nodes . But Elasticsearch is running fine .

Please help me hear how to resolve .

Regards  
Praveen

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [May 17, 2023, 2:02pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/17 "2023-05-17T14:02:57Z")

</div>

Are you able to click the "Add Action" button? or is it hidden? If you can click the button which Actions do you see? For example in my 7.x cluster I see: Email, Logging, Slack, Webhook, Index, Pagerduty, Jira. As far as the nohup error. If it is still causing a problem I'd suggest opening up a different question for that.

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [May 17, 2023, 5:38pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/18 "2023-05-17T17:38:37Z")

</div>

Hi Sullivan,

I am able to configure watcher and enable email notification from kibana but facing issue in action tab —. Email- —- while testing the mail - we are getting error as “Failed to send email notification.”

Any idea??

---

<div class="post-metadata">

**Author:** ![Wave](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wave/32/117242_2.png) [@Wave](https://discuss.elastic.co/u/Wave)\
**Post date:** [May 22, 2023, 9:05pm UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/19 "2023-05-22T21:05:34Z")

</div>

Not sure, if you could include any log or message on what error you are seeing that would be helpful.

---

<div class="post-metadata">

**Author:** ![Praveen\_kr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/praveen_kr/32/119997_2.png) [@Praveen\_kr](https://discuss.elastic.co/u/Praveen_kr)\
**Post date:** [May 24, 2023, 5:38am UTC](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291/20 "2023-05-24T05:38:52Z")

</div>

Hello Sullivan,

Thanks a lot! I am able to enable the watcher from kibana.

Challenge what am facing here is when I try to test mail from kibana - watcher- Add action— email - test email….

Error as “ failed to send email to [abcd@company.com](mailto:abcd@company.com)”

Tried checking in Elasticsearch logs - I don’t see any errors logs etc.

Could you please help me to troubleshoot this.

Regards,  
Praveen

[Next page](https://discuss.elastic.co/t/need-watcher-configuration-and-settings-elasticsearch-yml/330291.md?page=2)