Watcher Email configuration Issue

r.ganeshbabu · October 27, 2015, 6:21am

Hi,

I have configured watcher to Elasticsearch to get Alert & Notifications and I have setup Email configuration in yml file as shown below.

watcher.actions.email.service.account:
company_account:
profile: company
smtp:
auth: true
starttls.enable: true
host: smarthost.company.com
port: 25
user: ganeshbabu.ramamoorthy@company.com
password: password

But I am getting error in log file as,

[2015-10-27 02:09:37,184][INFO ][watcher ] [server_MASTER] starting watch service...
[2015-10-27 02:09:37,765][INFO ][watcher ] [server_MASTER] watch service has started
[2015-10-27 02:10:38,130][ERROR][watcher.actions.email ] [server_MASTER] failed to execute action [event_critical_watch/email_admin]
org.elasticsearch.watcher.actions.email.service.EmailSettingsException: cannot find default email account as no accounts have been configured
at org.elasticsearch.watcher.actions.email.service.Accounts.account(Accounts.java:71)
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:74)
at org.elasticsearch.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:67)
at org.elasticsearch.watcher.actions.ActionWrapper.execute(ActionWrapper.java:104)
at org.elasticsearch.watcher.execution.ExecutionService.executeInner(ExecutionService.java:379)
at org.elasticsearch.watcher.execution.ExecutionService.execute(ExecutionService.java:271)
at org.elasticsearch.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:417)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Watcher Email configuration should be added in all nodes right?

Please guide us to fix this issue and it would be very helpful.

Thanks,
Ganeshbabu R

spinscale · October 27, 2015, 10:18am

Hey,

I just tested your example configuration and found this message in the logs, when starting elasticsearch

[2015-10-27 11:16:02,375][INFO ][watcher.actions.email.service] [Sigyn] default account set to [company_account]

This means that a default account has been set and above error message should not occur. Can you restart your node and see if you find a similar message on startup? If not, please recheck the indendation of your configuration, I assume there might be a slight issue and thus the parsing fails.

If both do not work, please provide the exact configuration including indentation as well as the elasticsearch and watcher version, so we can try to reproduce this. Thanks!

--Alex

r.ganeshbabu · October 27, 2015, 11:48am

Thanks for your response alex.

We are using Elasticsearch version as 1.7.2 & watcher 1.0
Still I am facing the same issue.

Please find the watcher creation details, index mapping setup details & error log details.

Watcher creation

PUT /_watcher/watch/event_critical_watch
{
"trigger": {
"schedule": {
"interval": "60s"
}
},
"input": {
"search": {
"request": {
"indices": [
"event"
],
"body": {
"query": {
"match": {
"eventCategory": "CRITICAL"
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"email_admin": {
"email": {
"to": "ganeshbabu.ramamoorthy@company.com",
"subject": "{{ctx.watch_id}} executed",
"body": "{{ctx.watch_id}} executed with {{ctx.payload.hits.total}} hits"
}
}
}
}

Sample index event creation

PUT event
{
"settings" : {
"number_of_shards" : 1
},
"mappings" :{
"event": {
"properties": {
"eventId": {
"type": "integer"
},
"eventName": {
"type": "string"
},
"eventDescription": {
"type": "string"
},
"eventCategory": {
"type": "string"
},
"eventType": {
"type": "string"
}

}
}

}
}

Sample index data creation for event

PUT /event/event/1
{
"eventId" : 1,
"eventName" : "3 failed login attempts",
"eventDescription" : "System has detected 3 failed login attempts",
"eventCategory" : "CRITICAL",
"eventType" : "LOG"
}

After restarted the Elasticsearch cluster

[2015-10-27 07:20:43,489][INFO ][shield.license ] [server_MASTER] enabling license for [shield]
[2015-10-27 07:20:43,490][INFO ][license.plugin.core ] [server_MASTER] license for [shield] - valid
[2015-10-27 07:20:43,510][INFO ][license.plugin.core ] [server_MASTER] license for [watcher] - valid
[2015-10-27 07:20:43,647][INFO ][gateway ] [server_MASTER] recovered [34] indices into clu
[2015-10-27 07:20:52,547][INFO ][watcher ] [server_MASTER] starting watch service...
[2015-10-27 07:20:53,010][INFO ][watcher ] [server_MASTER] watch service has started
[2015-10-27 07:21:53,229][ERROR][watcher.actions.email ] [server_MASTER] failed to execute action [event_critical_watch/email_admin]
org.elasticsearch.watcher.actions.email.service.EmailSettingsException: cannot find default email account as no accounts have been configured
at org.elasticsearch.watcher.actions.email.service.Accounts.account(Accounts.java:71)
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:74)
at org.elasticsearch.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:67)

Watcher configuration in YML File

watcher.actions.email.service.account:
company_account:
profile: company
smtp:
auth: true
starttls.enable: true
host: smarthost.company.com
port: 25
user: ganeshbabu.ramamoorthy@company.com
password: password

NOTE:- Company is an example hostname instead of that we will be using our smtp server name

Please let me know if i need to make any changes in setting/yml level.

Regards,
Ganeshbabu R

warkolm · October 28, 2015, 2:23am

Make sure things are tabbed/spaced correctly, Watcher configs can trip on that sometimes.

r.ganeshbabu · October 28, 2015, 7:54am

Thanks Mark

I made a mistake in watcher configs (Not given proper tab/space).

After I restarted Elasticsearch I have checked the log what I found is,

[2015-10-28 03:41:13,761][INFO ][watcher.actions.email.service] [server_MASTER] default account set to [company_account]
[2015-10-28 03:41:13,867][INFO ][shield.license ] [server_MASTER] enabling license for [shield]
[2015-10-28 03:41:13,868][INFO ][license.plugin.core ] [server_MASTER] license for [shield] - valid
[2015-10-28 03:41:13,884][INFO ][license.plugin.core ] [server_MASTER] license for [watcher] - valid
[2015-10-28 03:41:13,997][INFO ][gateway ] [server_MASTER] recovered [36] indices into cluster_state
[2015-10-28 03:41:17,047][INFO ][cluster.service ] [server_MASTER] added {[server_DATA][IY2HI3JmYW3rqUl23YQ][server.enterprisenet.org][inet[/localhost:9260]]{max_local_storage_nodes=1, master=false},}, reason: zen-disco-receive(join from node[[server_DATA][IY2HI3JmSYW3rqUltV23YQ][server.enterprisenet.org][inet[/localhost:9260]]{max_local_storage_nodes=1, master=false}])
[2015-10-28 03:41:19,167][INFO ][watcher ] [server_MASTER] starting watch service...
[2015-10-28 03:41:19,569][INFO ][watcher ] [server_MASTER] watch service has started
[2015-10-28 03:41:22,374][INFO ][cluster.service ] [server_MASTER] added {[server_CLIENT][psOYfNHXS_yJ3C5gFE7w][server.enterprisenet.org][inet[/localhost:9210]]{max_local_storage_nodes=1, data=false, master=false},}, reason: zen-disco-receive(join from node[[server_CLIENT][psOYfNHXS_yHiJ3C5gFE7w][server.enterprisenet.org][inet[/localhost:9210]]{max_local_storage_nodes=1, data=false, master=false}])
[2015-10-28 03:42:19,827][ERROR][watcher.actions.email ] [server_MASTER] failed to execute action [event_critical_watch/email_admin]
org.elasticsearch.watcher.actions.email.service.EmailException: failed to send email with subject [event_critical_watch executed] via account [company_account]
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:86)
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:78)
at org.elasticsearch.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:67)
at org.elasticsearch.watcher.actions.ActionWrapper.execute(ActionWrapper.java:104)
at org.elasticsearch.watcher.execution.ExecutionService.executeInner(ExecutionService.java:379)
at org.elasticsearch.watcher.execution.ExecutionService.execute(ExecutionService.java:271)
at org.elasticsearch.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:417)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: com.sun.mail.util.MailConnectException: Couldn't connect to host, port: smarthost.company.com, 25; timeout -1;
nested exception is:
java.net.UnknownHostException: smarthost.company.com
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2054)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:697)
at javax.mail.Service.connect(Service.java:364)
at org.elasticsearch.watcher.actions.email.service.Account.send(Account.java:99)
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:84)
... 9 more
Caused by: java.net.UnknownHostException: smarthost.company.com
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at com.sun.mail.util.SocketFetcher.createSocket(SocketFetcher.java:329)
at com.sun.mail.util.SocketFetcher.getSocket(SocketFetcher.java:236)
at com.sun.mail.smtp.SMTPTransport.openServer(SMTPTransport.java:2020)
... 13 more

Please let me know whether I need to make any configuration for connecting SMTP Server from my server.

Thanks,
Ganeshbabu R

spinscale · October 28, 2015, 7:57am

hey,

the error message clearly (well, clearly is tricky in the land of java exceptions... states, that you configured a host that could not be found, so this is a config problem that occurs when trying to sent a mail.

java.net.UnknownHostException: smarthost.company.com

r.ganeshbabu · October 28, 2015, 8:02am

Thank you alex

r.ganeshbabu · October 30, 2015, 9:55am

@warkolm @spinscale

I configured the dev server with smarthost.company.com (HOST). Now I didn't get the error as couldn't connect host:Port after the configuration.

But now I got authentication exception when trying to send a mail.

[2015-10-30 05:33:44,834][ERROR][watcher.actions.email ] [server_MASTER] failed to execute action [event_critical_watch/email_admin]
org.elasticsearch.watcher.actions.email.service.EmailException: failed to send email with subject [event_critical_watch executed] via account [company_account]
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:86)
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:78)
at org.elasticsearch.watcher.actions.email.ExecutableEmailAction.execute(ExecutableEmailAction.java:67)
at org.elasticsearch.watcher.actions.ActionWrapper.execute(ActionWrapper.java:104)
at org.elasticsearch.watcher.execution.ExecutionService.executeInner(ExecutionService.java:379)
at org.elasticsearch.watcher.execution.ExecutionService.execute(ExecutionService.java:271)
at org.elasticsearch.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:417)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.mail.AuthenticationFailedException: No authentication mechanisms supported by both server and client
at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:818)
at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:728)
at javax.mail.Service.connect(Service.java:364)
at org.elasticsearch.watcher.actions.email.service.Account.send(Account.java:99)
at org.elasticsearch.watcher.actions.email.service.InternalEmailService.send(InternalEmailService.java:84)
... 9 more

Please help me to resolve it would be very helpful.

Thanks,
Ganeshbabu R

Enniu_51 · October 30, 2015, 10:03am

It seems like the email server of your company @company.com don't need authentication, but in your config:

smtp:
    auth: true

claims authentication needed.

r.ganeshbabu · November 3, 2015, 7:53am

Thanks for your response @Enniu_51

I have created new watcher

PUT /_watcher/watch/cluster_health_watch
{
"trigger" : {
"schedule" : { "interval" : "30s" }
},
"input" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : port,
"path" : "/_cluster/health"
}
}
},
"condition" : {
"compare" : {
"ctx.payload.status" : { "eq" : "green" }
}
},
"actions" : {
"send_email" : {
"email" : {
"to" : "ganeshbabu.ramamoorthy@company.com",
"subject" : "Cluster Status Health",
"body" : "Cluster status is Green"
}
}
}
}

When I checked watcher history i am getting the following results,

{
"took": 61,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 27,
"max_score": 1,
"hits": [
{
"_index": ".watch_history-2015.11.03",
"_type": "watch_record",
"_id": "cluster_health_watch_0-2015-11-03T07:18:09.701Z",
"_score": 1,
"_source": {
"watch_id": "cluster_health_watch",
"state": "execution_not_needed",
"trigger_event": {
"type": "schedule",
"triggered_time": "2015-11-03T07:18:09.701Z",
"schedule": {
"scheduled_time": "2015-11-03T07:18:09.395Z"
}
},
"input": {
"http": {
"request": {
"scheme": "http",
"host": "localhost",
"port": port,
"method": "get",
"path": "/_cluster/health",
"params": {},
"headers": {}
}
}
},
"condition": {
"compare": {
"ctx.payload.status": {
"eq": "green"
}
}
},
"messages": [],
"result": {
"execution_time": "2015-11-03T07:18:09.701Z",
"execution_duration": 105,
"input": {
"type": "http",
"status": "success",
"payload": {},
"http": {
"request": {
"host": "localhost",
"port": port,
"scheme": "http",
"method": "get",
"path": "/_cluster/health"
},
"status_code": 401
}
},
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
"ctx.payload.status": null
}
}
},
"actions": []
}
}
},

Why ctx.payload.status become "null" and action block shown as empty.

But I can see the count is increasing based on the interval(30s).

Why I didn't any alert mail from the server.

Is watcher will only work for negative impact but i am creating watcher with positive effect

Will it work or not?

Any help would be much appreciated.

Thanks,
Ganeshbabu R

Enniu_51 · November 4, 2015, 1:42am

Watcher will send email when ctx.payload.status is green, but you status in null.

I have no idea about why your ctx.payload.status is null, because I never use a input as yours.

You may @ official guys and they may help you.

r.ganeshbabu · November 4, 2015, 4:46am

Thanks for your response @Enniu_51

r.ganeshbabu · November 4, 2015, 4:53am

Hi @warkolm

I have created new watcher

PUT /_watcher/watch/cluster_health_watch
{
"trigger" : {
"schedule" : { "interval" : "30s" }
},
"input" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : port,
"path" : "/_cluster/health"
}
}
},
"condition" : {
"compare" : {
"ctx.payload.status" : { "eq" : "green" }
}
},
"actions" : {
"send_email" : {
"email" : {
"to" : "ganeshbabu.ramamoorthy@company.com",
"subject" : "Cluster Status Health",
"body" : "Cluster status is Green"
}
}
}
}

When I checked watcher history i am getting the following results,

{
"took": 61,
"timed_out": false,
"_shards": {
"total": 1,
"successful": 1,
"failed": 0
},
"hits": {
"total": 27,
"max_score": 1,
"hits": [
{
"_index": ".watch_history-2015.11.03",
"_type": "watch_record",
"_id": "cluster_health_watch_0-2015-11-03T07:18:09.701Z",
"_score": 1,
"_source": {
"watch_id": "cluster_health_watch",
"state": "execution_not_needed",
"trigger_event": {
"type": "schedule",
"triggered_time": "2015-11-03T07:18:09.701Z",
"schedule": {
"scheduled_time": "2015-11-03T07:18:09.395Z"
}
},
"input": {
"http": {
"request": {
"scheme": "http",
"host": "localhost",
"port": port,
"method": "get",
"path": "/_cluster/health",
"params": {},
"headers": {}
}
}
},
"condition": {
"compare": {
"ctx.payload.status": {
"eq": "green"
}
}
},
"messages": [],
"result": {
"execution_time": "2015-11-03T07:18:09.701Z",
"execution_duration": 105,
"input": {
"type": "http",
"status": "success",
"payload": {},
"http": {
"request": {
"host": "localhost",
"port": port,
"scheme": "http",
"method": "get",
"path": "/_cluster/health"
},
"status_code": 401
}
},
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
"ctx.payload.status": null
}
}
},
"actions": []
}
}
},

Why ctx.payload.status become "null" and action block shown as empty.

But I can see the count is increasing based on the interval(30s).

Why I didn't any alert mail from the server.

Is watcher will only work for negative scenario but I created watcher with positive scenario

will it work?

Please help me to resolve this issue.

Thanks,
Ganeshbabu R

ksoucy · February 29, 2016, 5:05pm

Hi, your status code shows your cluster health check is not authorized to execute ("status_code": 401). Did you by any chance install and enable Shield on your cluster? If so, you have to include the "auth" parameter in your input section (including an id and password for an account configured in Shield that is part of the Watcher role).

r.ganeshbabu · March 1, 2016, 4:57am

Thanks for your response @ksoucy

After I include the auth details in the input section I didn't get any errors and It's worked good.

Regards,
Ganeshbabu R

vbi517 · July 12, 2016, 10:55am

Hi Ganeshprabhu, Could u please elaborate about including auth details in the input section....bcs i do facing same issue...

r.ganeshbabu · July 12, 2016, 11:10am

Hi @vbi517

You need to include the auth details in the input section as like below,

PUT /_watcher/watch/cluster_health_watch
{
"trigger" : {
"schedule" : { "interval" : "60s" }
},
"input" : {
"http" : {
"request" : {
"host" : "10.7.146.25",
"port" : 9200,
"path" : "/_cluster/health",
> "auth": {

      "basic": {
        "username": "esadmin",
        "password": "xxxxxxx"
      }
    }

}
}

},

vbi517 · July 12, 2016, 11:22am

Thanks Ganeshprabhu for quick valuable inputs...

Could you please let me know...where do i add auth in following input i.e.
curl -XPUT 'http://localhost:9200/_watcher/watch/log_error_watch' -d '{
"trigger" : { "schedule" : { "interval" : "10s" } },
"input" : {
"search" : {
"request" : {
"indices" : [ "logs" ],
"body" : {
"query" : {
"match" : { "message": "error" }
}
}
}
}
},

vbi517 · July 12, 2016, 11:31am

HI Ganeshprabhu,

is following edit valid one...

"input" : {
"search" : {
"request" : {
"indices" : [ "logs" ],
"body" : {
"query" : {
"match" : { "message": "error" }
}
}
"auth": {
"basic": {
"username": "vijaykumar@xyz.com",
"password": "xyz123"
}
}
}
}
},

r.ganeshbabu · July 12, 2016, 11:37am

@vbi517

What is the response after you adding the authentication details in the input section?