Need Watcher configuration and settings ElasticSearch yml

Hi Team,
,

Anyone one Could you please help me with the watcher configuration elasticsearch yml setup as we have 13 nodes I need to add the watcher settings to send a mail alert (outlook). Challenges what am facing here. It is completely secured project and need to add watcher settings to get email alert to my client. So I referred many pages in elastic I was facing issue adding setup in Elasticsearch yml file.

Elasticsearch version 7.17 version we are using
Totally 13 nodes ( master node - 3 , Data node - 10)

Questions — is it possible to test the watcher in one master node and one data node to enable the email alert ???

And In our environment we have default port 25 smtp

Hi @Praveen_kr,

You'll need to add the settings to all your nodes for it to start working. That can be a big ask so often I'll save those changes for when I'm going to do a rolling restart anyway. Here is the documentation that should help out with watcher configuration.

Hi Sullivan,

Thanks for your support. I had configured the settings for outlook but password am missing to store in secure way. I didn’t added password in the watcher settings. How to proceed further. I need to add the password in all the nodes to move head. Till that watcher is disabled in kibana I can’t enable no option .

Could you please suggest how to add my outlook password to test the watcher email is working ??

Want to correct something from first reply, you'll need to add those changes to all the nodes, but I don't think you'll need to do a restart for the email settings. If you follow the link to the documentation provided on my original reply you'll see the following:

smtp.secure_password is a secure and reloadable setting.
Almost all the smtp settings are also dynamic, which means you can set them through Kibana.

As far as configuring it securely, see here for using secure settings.

Hi Sullivan,

This is my settings in yml file

Xpack.watcher.enabled: true
Xpack.notification.email.account:
Outlook_account:
Profile: outlook
SMTP:
auth: true
Starttls.enable: true
Host: smtp-mail.outlook.com
Port: 25
User: xxxxx@company.com

Anything is missing in configuration please let me know.
Password I didn’t mention but am not sure how to add here after user in the settings.

Please suggest me

Regards,
Praveen

This is what I'd use regarding watcher in your elasticsearch.yml file:

xpack.watcher.enabled: true

From dev tools in kibana I'd set the following cluster settings since they are dynamic:

xpack.notification.email.account.outlook_account.profile: outlook
xpack.notification.email.account.outlook_account.smtp.auth: true
xpack.notification.email.account.outlook_account.smtp.starttls.enable: true
xpack.notification.email.account.outlook_account.smtp.host: smtp-mail.outlook.com
xpack.notification.email.account.outlook_account.smtp.user: xxxxx@company.com

You could totally have them display nested didn't do that here.

You will then HAVE to add the setting "xpack.notification.email.account.outlook_account.smtp.secure_password" to the elasticsearch keystore for things to work securely. If that isn't added there it's never going to work. Please see the following documentation about doing that: elasticsearch-keystore docs.

Good luck

Hi Sullivan,

Thanks a lot for ur help . Challenges what am facing adding outlook password in yml file.
I am not sure how to add password in Keystore I went through documentation but still I have concern regarding those things.

Example : In kibana under stack management- watcher- threshold alert - action - email ——-this is not enable yet. I tried adding watcher settings in yml file on 3 master nodes and restarted the elasticsearch still I don’t see the enable in kibana. Also lot of security concern I have to go step by step as explained earlier . We have 10 data nodes and 3 master nodes . In all these nodes I need to add watcher outlook settings and restart the service ???

Please help me how will I proceed further.

Regards,
Praveen

Please read my previous posts because I feel like I've already answered these questions, but just to be clear:

Yes, you'll need to enable watcher and do a rolling-restart.

xpack.watcher.enabled: true

on all the nodes (master, data, etc). Just doing some will NOT work.
Can you tell me what your security concerns are? Without more information storing sensitive information in the elastic keystore is the best practice for handling that.

Hi Sullivan,
My concern here adding the password in a keyword tool. Is it mandatory adding outlook password in a Keystore tool???

I don’t know how to create the Keystore from
Beginnings and adding the outlook password?

Regards,
Praveen

Ok, lets take a step back here and talk about what you're trying to accomplish. Reading between the lines it sounds like you might be connecting a fully functioning email address up with watcher. While that would probably work often a noreply address is used instead (noreply@example.com). Using an account like could limit the exposure if somehow the username and password were compromised.

Another option is if you just need some kind of alert, watcher supports other actions besides email. For example: webhook, slack, pagerduty. Perhaps one of those would work just as well.

Finally lets talk about the options to set variables in elastic.

  1. You could set the password via command line. This is not secure at all because any other user logged into the system could see the password.
  2. Setting the password directly in the elasticsearch.yml file. This is a little bit better assuming you have your file permissions set correctly, but you are still dependent upon the OS properly limiting read access to the yml file.
  3. Using the keystore. The keystore was made for just these purposes. It will keep secure settings encrypted until needed by the program. This is the most secure option of the three.

To answer your questions about the keystore, no it is not mandatory could use option 1 or 2 if you wanted to but those aren't better. The keystore should be made automatically by elastic and should be in the same directory as your elasticsearch.yml file.

Final thought is please read and become familiar with the links provided. The answers to many of your questions will be found in the documentation, and while I try my best to answer and help this community the ultimate responsibility for your cluster falls on you and your understanding of the issues. Having said that I'm happy to help where I can, but can only do so much.

Hi Sullivan,

I have configured watcher in yml file all 3 master nodes , but still don’t see the enable option in kibana .

I.e Kibana - stack management - watcher - create watcher - add condition - add action —— here I don’t see the email enabled.

You need to configure the watcher settings in all the nodes in the cluster. Just master nodes won't cut it. Must enable on data nodes as well. You probably won't see an enable option in kibana, but you will be able to send emails from watcher once correctly configured.

Sure Sullivan will configure same for all remaining nodes.

In Elasticsearch yml file I have mentioned all the outlook settings as per the elastic documentation. password I have created Keystore..

Question——— under the user password : can I mention $ {outlook.smtp.password} the same as below ?????

smtp: auth: true starttls.enable: true host: smtp.office365.com port: 587 user: @outlook.com password: ${outlook.smtp.password}

Regards,
Praveen

I believe you won't have to put the password part in the elasticsearch.yml file because it will look in the keystore for it. You will need make sure the password is in each keystore as well.

Hello Sullivan,

Appreciate your support. I am working on data nodes to add the watcher setting remaining 10 nodes. If I face any issue challenging will let you know by tomorrow.

Regards,
Praveen

Hi Sullivan,

I have restarted all the data nodes and master nodes but the don’t see watcher enable in kibana.
Stack management - Watcher- create threshold alert- Add action - here I don’t see enable still.
Could you please help me out.

Also on see log message nohup.error. With 0 messages in all the nodes . But Elasticsearch is running fine .

Please help me hear how to resolve .

Regards
Praveen

Are you able to click the "Add Action" button? or is it hidden? If you can click the button which Actions do you see? For example in my 7.x cluster I see: Email, Logging, Slack, Webhook, Index, Pagerduty, Jira. As far as the nohup error. If it is still causing a problem I'd suggest opening up a different question for that.

Hi Sullivan,

I am able to configure watcher and enable email notification from kibana but facing issue in action tab —. Email- —- while testing the mail - we are getting error as “Failed to send email notification.”

Any idea??

Not sure, if you could include any log or message on what error you are seeing that would be helpful.

Hello Sullivan,

Thanks a lot! I am able to enable the watcher from kibana.

Challenge what am facing here is when I try to test mail from kibana - watcher- Add action— email - test email….

Error as “ failed to send email to abcd@company.com

Tried checking in Elasticsearch logs - I don’t see any errors logs etc.

Could you please help me to troubleshoot this.

Regards,
Praveen