Elasticsearch Alert Email not working - Watcher

david-vazquez · May 6, 2020, 5:55pm

Dear all,

I configured SMTP following the guide https://www.elastic.co/guide/en/elasticsearch/reference/7.6/actions-email.html#configuring-email

My configuration is as follows:
xpack.notification.email.account:
gmail_account:
profile: gmail
smtp:
auth: true
starttls.enable: true
host: smtp.gmail.com
port: 587
user: myemail@gmail.com

Then i run the command bin/elasticsearch-keystore add xpack.notification.email.account.gmail_account.smtp.secure_password

I got an error when I tryed to send a test email, which I will show in next message.

I tryed it with a outlook account and with gmail account. Also I allowed less secured apps in Gmail.

I see one error of bad username, I don´t know why because usermail is right, and when i run the command I introduced the right password.

Any idea?

Thank you very much

david-vazquez · May 6, 2020, 5:55pm

[2020-05-06T19:49:20,808][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [localhost.l                                                                                                                                                                                                                                             ocaldomain] failed to execute action [_inlined_/email_1]
javax.mail.MessagingException: failed to send email with subject [Watch [Alerta                                                                                                                                                                                                                                              uso de CPU elevado] has exceeded the threshold] via account [gmail_account]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(                                                                                                                                                                                                                                             EmailService.java:171) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(                                                                                                                                                                                                                                             EmailService.java:163) ~[?:?]
        at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.e                                                                                                                                                                                                                                             xecute(ExecutableEmailAction.java:76) ~[?:?]
        at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(Ac                                                                                                                                                                                                                                             tionWrapper.java:164) [x-pack-core-7.6.0.jar:7.6.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInn                                                                                                                                                                                                                                             er(ExecutionService.java:534) [x-pack-watcher-7.6.0.jar:7.6.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(Ex                                                                                                                                                                                                                                             ecutionService.java:320) [x-pack-watcher-7.6.0.jar:7.6.0]
        at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportEx                                                                                                                                                                                                                                             ecuteWatchAction$1.doRun(TransportExecuteWatchAction.java:159) [x-pack-watcher-7                                                                                                                                                                                                                                             .6.0.jar:7.6.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(Abstrac                                                                                                                                                                                                                                             tRunnable.java:37) [elasticsearch-7.6.0.jar:7.6.0]
        at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecu                                                                                                                                                                                                                                             tionTask.run(ExecutionService.java:627) [x-pack-watcher-7.6.0.jar:7.6.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreserv                                                                                                                                                                                                                                             ingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.0.jar:7.6.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.                                                                                                                                                                                                                                             java:1128) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor                                                                                                                                                                                                                                             .java:628) [?:?]
        at java.lang.Thread.run(Thread.java:830) [?:?]
Caused by: javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Pass                                                                                                                                                                                                                                             word not accepted. Learn more at
535 5.7.8  https://support.google.com/mail/?p=BadCredentials g6sm3915193wrw.34 -                                                                                                                                                                                                                                              gsmtp

        at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransp                                                                                                                                                                                                                                             ort.java:947) ~[?:?]
        at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:858)                                                                                                                                                                                                                                              ~[?:?]
        at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:76                                                                                                                                                                                                                                             2) ~[?:?]
        at javax.mail.Service.connect(Service.java:342) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.lambda$exe                                                                                                                                                                                                                                             cuteConnect$2(Account.java:161) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:554                                                                                                                                                                                                                                             ) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.executeCon                                                                                                                                                                                                                                             nect(Account.java:160) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.Account.send(Accou                                                                                                                                                                                                                                             nt.java:119) ~[?:?]
        at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(                                                                                                                                                                                                                                             EmailService.java:169) ~[?:?]
        ... 12 more
[root@localhost elasticsearch]#

david-vazquez · May 6, 2020, 10:27pm

It´s going crazy... If I swap the configuration between outlook and gmail, it always throw an error regarding Gmail even if Im using outlook in the configuration...

I don´t know how to solve this problem.

Is there any way to just use a "non-official" email? just something like no-reply@watcher.com or alerts@watcher.com , being generated automatically by Elastic. I tryed Wazuh a long time ago and it could do it, without need to introduce your SMTP configuration.

Luca_Belluccini · May 6, 2020, 11:34pm

Hello @david-vazquez

Sorry to ask but:

Did you create a dedicated password for your Gmail account at this link? You cannot use the normal Gmail credentials, especially if you are using 2FA.
If you have multiple Elasticsearch nodes, did add the command bin/elasticsearch-keystore add xpack.notification.email.account.gmail_account.smtp.secure_password on all nodes?

david-vazquez · May 7, 2020, 11:23am

Hello @Luca_Belluccini,

Im not using 2FA. Also didn´t create Gmail dedicated password. I will try it. Anyway, I tryed with gmail just because it was not working with Outlook. I dont care what u se, just would like to work
I have a single node with Elasticsearch, so this is not the problem.

The thing is If I want to configure it again with outlook, Gmail error is always showed. Configuration is not working for some reason, I dont know whats happening now.

Thanks

Luca_Belluccini · May 7, 2020, 11:34am

We need to check the settings as the updates are not taken into account.

david-vazquez · May 7, 2020, 2:53pm

Hello Luca,

I followed setting as I said in first message. I mean, just copied and paste the configuration thats showed in the guide, changuind the value of the user account.
What exactly do you need to check?

Thank you

Luca_Belluccini · May 7, 2020, 6:07pm

Hello @david-vazquez, I'm trying to help.

There is for sure a misconfiguration.

What do you mean? The SMTP server "can" allow a non-existing email, but not all SMTP servers allow that.

If you're copy-pasting the configuration, ensure to keep the formatting of the yaml file or it can happen that some of the configurations you've done are not taken into account.
Try to send the following request and share the response
```
GET _cluster/settings?flat_settings&include_defaults
```
If you're configuring multiple email accounts, on the watcher you need to specify the profile/account to be used, or the default one will be used. For the sake of avoiding errors, share at least the action section of your Watch.

I have literally your same configuration on a test cluster and I can correctly send the emails via GMail.

david-vazquez · May 7, 2020, 6:38pm

I know you are trying to help, thank you very much.

Finally it´s working... to be honest, I don´t know what I do different from yesterday. Maybe it was problem about format. I´m using Gmail, exactly the same than yesterday, and today is working

We can say sometimes it´s just magic.

Anyway, thank you very much for helping me.

Luca_Belluccini · May 7, 2020, 6:43pm

No problem I'm glad it's working now!

Jalpesh1 · May 25, 2020, 2:17pm

Hi Team,

I am facing same issue.

Please find below output for my system.

GET _cluster/settings?flat_settings&include_defaults

{
"persistent" : { },
"transient" : { },
"defaults" : {
"action.auto_create_index" : "true",
"action.destructive_requires_name" : "false",
"action.search.shard_count.limit" : "9223372036854775807",
"xpack.monitoring.collection.interval" : "10s",
"xpack.monitoring.collection.ml.job.stats.timeout" : "10s",
"xpack.monitoring.collection.node.stats.timeout" : "10s",
"xpack.monitoring.elasticsearch.collection.enabled" : "true",
"xpack.monitoring.enabled" : "true",
"xpack.monitoring.history.duration" : "168h",
"xpack.notification.email.account.gmail_account.profile" : "gmail",
"xpack.notification.email.account.gmail_account.smtp.auth" : "true",
"xpack.notification.email.account.gmail_account.smtp.host" : "smtp.gmail.com",
"xpack.notification.email.account.gmail_account.smtp.port" : "587",
"xpack.notification.email.account.gmail_account.smtp.starttls.enable" : "true",
"xpack.notification.email.account.gmail_account.smtp.user" : "qktestalert@gmail.com",
"xpack.notification.email.default_account" : "",
"xpack.notification.email.html.sanitization.allow" : [
"body",
"head",
"_tables",
"_links",
"_blocks",
"_formatting",
"img:embedded"
],
"xpack.notification.email.html.sanitization.disallow" : ,
"xpack.notification.email.html.sanitization.enabled" : "true",
"xpack.notification.jira.default_account" : "",
"xpack.notification.pagerduty.default_account" : "",
"xpack.notification.reporting.interval" : "15s",
"xpack.notification.reporting.retries" : "40",
"xpack.notification.reporting.warning.enabled" : "true",
"xpack.notification.slack.default_account" : "",
"xpack.rollup.enabled" : "true",
"xpack.rollup.task_thread_pool.queue_size" : "4",
"xpack.rollup.task_thread_pool.size" : "4",
"xpack.security.audit.enabled" : "false",
"xpack.security.audit.logfile.emit_node_host_address" : "false",
"xpack.security.audit.logfile.emit_node_host_name" : "false",
"xpack.security.audit.logfile.emit_node_id" : "true",
"xpack.security.audit.logfile.emit_node_name" : "false",
"xpack.security.audit.logfile.events.emit_request_body" : "false",
"xpack.security.audit.logfile.events.exclude" : ,
"xpack.security.audit.logfile.events.include" : [
"ACCESS_DENIED",
"ACCESS_GRANTED",
"ANONYMOUS_ACCESS_DENIED",
"AUTHENTICATION_FAILED",
"CONNECTION_DENIED",
"TAMPERED_REQUEST",
"RUN_AS_DENIED",
"RUN_AS_GRANTED"
],
"xpack.security.authc.anonymous.authz_exception" : "true",
"xpack.watcher.actions.bulk.default_timeout" : "",
"xpack.watcher.actions.index.default_timeout" : "",
"xpack.watcher.bulk.actions" : "1",
"xpack.watcher.bulk.concurrent_requests" : "0",
"xpack.watcher.bulk.flush_interval" : "1s",
"xpack.watcher.bulk.size" : "1mb",
"xpack.watcher.enabled" : "true",
"xpack.watcher.encrypt_sensitive_data" : "false",
"xpack.watcher.execution.default_throttle_period" : "5s",
"xpack.watcher.execution.scroll.size" : "0",
"xpack.watcher.execution.scroll.timeout" : "",
"xpack.watcher.history.cleaner_service.enabled" : "true",
"xpack.watcher.index.rest.direct_access" : "",
"xpack.watcher.input.search.default_timeout" : "",
"xpack.watcher.internal.ops.bulk.default_timeout" : "",
"xpack.watcher.internal.ops.index.default_timeout" : "",
"xpack.watcher.internal.ops.search.default_timeout" : "",
"xpack.watcher.stop.timeout" : "30s",
"xpack.watcher.thread_pool.queue_size" : "1000",
"xpack.watcher.thread_pool.size" : "20",
"xpack.watcher.transform.search.default_timeout" : "",
"xpack.watcher.trigger.schedule.ticker.tick_interval" : "500ms",
"xpack.watcher.watch.scroll.size" : "0"
}
}

Also find below logs:

[2020-05-25T19:19:56,581][WARN ][o.e.x.w.e.ExecutionService] [JALPESH] failed to execute watch [9b8f79c8-11c1-427d-97bf-ea8f7bd184f9]
[2020-05-25T19:20:56,641][WARN ][o.e.x.w.e.ExecutionService] [JALPESH] failed to execute watch [9b8f79c8-11c1-427d-97bf-ea8f7bd184f9]
[2020-05-25T19:24:59,153][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [JALPESH] failed to execute action [9b8f79c8-11c1-427d-97bf-ea8f7bd184f9/email_1]
javax.mail.MessagingException: failed to send email with subject [Watch [test4] has exceeded the threshold] via account [gmail_account]
at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:171) ~[?:?]
at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:163) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:76) ~[?:?]
at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:164) [x-pack-core-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:534) [x-pack-watcher-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:320) [x-pack-watcher-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:421) [x-pack-watcher-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$$Lambda$4674/1721056180.run(Unknown Source) [x-pack-watcher-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:627) [x-pack-watcher-7.6.2.jar:7.6.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.2.jar:7.6.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_40]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_40]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_40]
Caused by: javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8 https://support.google.com/mail/?p=BadCredentials v17sm13696968pfc.190 - gsmtp

at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:947) ~[?:?]
at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:858) ~[?:?]
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:762) ~[?:?]
at javax.mail.Service.connect(Service.java:342) ~[?:?]
at org.elasticsearch.xpack.watcher.notification.email.Account.lambda$executeConnect$2(Account.java:161) ~[?:?]
at org.elasticsearch.xpack.watcher.notification.email.Account$$Lambda$4974/246305874.run(Unknown Source) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_40]
at org.elasticsearch.xpack.watcher.notification.email.Account.executeConnect(Account.java:160) ~[?:?]
at org.elasticsearch.xpack.watcher.notification.email.Account.send(Account.java:119) ~[?:?]
at org.elasticsearch.xpack.watcher.notification.email.EmailService.send(EmailService.java:169) ~[?:?]
... 12 more

Luca_Belluccini · May 25, 2020, 5:12pm

The error states the credentials are wrong.

Same suggestions done before on:

And ensure in the watch you specify the account gmail_account.

system · June 22, 2020, 5:14pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.